2021-07-10T09:06:44.772 INFO:root:teuthology version: 1.1.0-24e3280d 2021-07-10T09:06:44.774 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi/6262049 --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-minimum} --name trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi --owner scheduled_trociny@teuthology -- /home/teuthworker/archive/trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi/6262049/orig.config.yaml 2021-07-10T09:06:44.834 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-07-10T09:06:44.947 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi/6262049 branch: wip-mgolub-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: to.my.trociny@gmail.com first_in_suite: false job_id: '6262049' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-mgolub-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 92adad3411b7280fb159eb9d7304c654695beb8b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 92adad3411b7280fb159eb9d7304c654695beb8b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mgolub-testing sha1: 92adad3411b7280fb159eb9d7304c654695beb8b owner: scheduled_trociny@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 sha1: 92adad3411b7280fb159eb9d7304c654695beb8b sleep_before_teardown: 0 suite: rbd suite_branch: wip-mgolub-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 92adad3411b7280fb159eb9d7304c654695beb8b targets: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjifRisnkBt6E3gXwjqJegeGV3N9KQSLRR3AhOQa1kN790cPaqd8ZBjIE8kHyKSQyoidtLxSE6vJTf07GZRht4joRBX8QVlr98pCqRD+h4zIAlbByupFYseek67lfsGdjW4v8Wrg1ETeERn89rLLhS2njcgQS/ZIRlchOeOuUhjrE+yGbYr7uI0VXp1zbVUsd0OjCyzDJUtOp8bDehlbGPSYS5jmYzfsaH3vDUlWP9xV1GmvyAGJPOXvaaXD2spe9luaLYS55UoQF60M+eSKciQPP0rCNnJqKnpnuRoiuw2hvgrxIwE/TSIbQnBoOqNIKL/dzhhul0TiGitVxelXTZwbdCri2ioPElcJ4YfOru9Lq2+oz9Vpvhr+IuABmzym+5atHmyOII5ueQ9WGTHUoB81A4jfpUBF1DV1vfelvLayrTaFbFyAo7zJwqs5KPy8+/q2egbJtXmnvBF1a5nF/iF+tDfwnGKVSD5/pK9WU79Vi5pmc04F4Gc80yf1haF+0= smithi105.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpGytXlOFlN1ptltJGikwEmNTcon0D67LpimkU8fksWbJap5NMtH0HZq37iy6W2NttOaC/S3NQ2iZ1lyYqX052aPHMgH2IIs70+sUELig4BxOJj6fri+23DBkfXaM+nCQcFlrMvOmJyJiRWaG9MNFXAPOHIMJrBu3CadnSvkkfia/W7f2FxwYqb3Ky7xKf1DUw2lL6UVDYG90t0/aYIPvykDntRlFPBWO1SCQ8hY/ZZzR7ZiXShJ2xT0NiRYrMw1GOJ5cCWKTurL29gr/At8h27Y4hHY7W+3Y9JKap5O1a3rMAcwdrQ7+JEI6QwyxZxE6pzWy8yLbLCp1pdp+9MkF07TJTUCImxVWh/QpYOMYqcxaxCxS/pVorKn3kUi8QPobCLFMRpQGBZTMAXPdDD/ScmifnWnwFWiNedppxdMiF8oJN30gAhqzHg21kki+m2YP4R1fgKleUVKD+3LrYvOy5preWF6hj14c8/80znrRb4O/gKKsU5mchA/6dT2qgv1M= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology_branch: master teuthology_sha1: 24e3280d7d1fc156628af95b24b27a29e38b2766 timestamp: 2021-07-10_06:51:06 tube: smithi user: trociny verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-07-10T09:06:44.948 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa; will attempt to use it 2021-07-10T09:06:44.949 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa/tasks 2021-07-10T09:06:44.949 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-07-10T09:06:44.951 INFO:teuthology.task.internal:Checking packages... 2021-07-10T09:06:44.979 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '92adad3411b7280fb159eb9d7304c654695beb8b' 2021-07-10T09:06:44.979 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-10T09:06:44.980 INFO:teuthology.packaging:ref: None 2021-07-10T09:06:44.980 INFO:teuthology.packaging:tag: None 2021-07-10T09:06:44.980 INFO:teuthology.packaging:branch: wip-mgolub-testing 2021-07-10T09:06:44.981 INFO:teuthology.packaging:sha1: 92adad3411b7280fb159eb9d7304c654695beb8b 2021-07-10T09:06:44.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-mgolub-testing 2021-07-10T09:06:45.142 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-5861.g92adad34 2021-07-10T09:06:45.142 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-07-10T09:06:45.151 INFO:teuthology.task.internal:no buildpackages task found 2021-07-10T09:06:45.152 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-07-10T09:06:45.388 INFO:teuthology.task.internal:Saving configuration 2021-07-10T09:06:45.408 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-07-10T09:06:45.418 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-10T09:06:45.436 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-07-10 09:01:48.640101', 'description': '/home/teuthworker/archive/trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi/6262049', 'locked_by': 'scheduled_trociny@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK658YH7Adf66hLSb5NYJDefI1TciGrtbDrzFXk/5JulXBjJfwHJpFWs+lVVg9G9WNIA6boWIXMLi/e/xEAAqVs=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi062.front.sepia.ceph.com'} 2021-07-10T09:06:45.453 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-07-10 09:01:48.640081', 'description': '/home/teuthworker/archive/trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi/6262049', 'locked_by': 'scheduled_trociny@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL435BMndQ8LwL3cJsLcVFkiyd7bZ1jRc2FEBaKmYbIjLKvDeDZS4vhJTv7kqliy0c903nNdBg5Ji7aXjDWVqd4=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi105.front.sepia.ceph.com'} 2021-07-10T09:06:45.453 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-07-10T09:06:45.464 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-07-10T09:06:45.464 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-07-10T09:06:45.465 INFO:teuthology.run_tasks:Running task console_log... 2021-07-10T09:06:45.565 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f60eb2efc80>, signals=[15]) 2021-07-10T09:06:45.566 INFO:teuthology.run_tasks:Running task internal.connect... 2021-07-10T09:06:45.576 INFO:teuthology.task.internal:Opening connections... 2021-07-10T09:06:45.577 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:06:45.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:06:45.651 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:06:45.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:06:45.721 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-07-10T09:06:45.735 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2021-07-10T09:06:45.754 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2021-07-10T09:06:45.754 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2021-07-10T09:06:45.811 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2021-07-10T09:06:45.812 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8" 2021-07-10T09:06:45.812 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2021-07-10T09:06:45.812 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2021-07-10T09:06:45.812 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8" 2021-07-10T09:06:45.812 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2021-07-10T09:06:45.813 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 8" 2021-07-10T09:06:45.813 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2021-07-10T09:06:45.813 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-07-10T09:06:45.813 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2021-07-10T09:06:45.813 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-07-10T09:06:45.814 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-07-10T09:06:45.814 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-07-10T09:06:45.815 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-07-10T09:06:45.832 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2021-07-10T09:06:45.850 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2021-07-10T09:06:45.850 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2021-07-10T09:06:45.907 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2021-07-10T09:06:45.908 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="8" 2021-07-10T09:06:45.908 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2021-07-10T09:06:45.908 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2021-07-10T09:06:45.909 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="8" 2021-07-10T09:06:45.909 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el8" 2021-07-10T09:06:45.909 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 8" 2021-07-10T09:06:45.909 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2021-07-10T09:06:45.910 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-07-10T09:06:45.910 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2021-07-10T09:06:45.910 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-07-10T09:06:45.910 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-07-10T09:06:45.911 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-07-10T09:06:45.911 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2021-07-10T09:06:45.927 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-07-10T09:06:45.942 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-07-10T09:06:45.975 INFO:teuthology.task.internal:Checking for old test directory... 2021-07-10T09:06:45.975 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2021-07-10T09:06:45.978 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2021-07-10T09:06:45.994 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-07-10T09:06:46.011 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-07-10T09:06:46.012 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2021-07-10T09:06:46.037 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2021-07-10T09:06:46.063 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-07-10T09:06:46.108 INFO:teuthology.run_tasks:Running task kernel... 2021-07-10T09:06:46.126 DEBUG:teuthology.task.kernel:normalized config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'cluster1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'cluster1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'cluster1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'cluster2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.0': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.1': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.2': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.3': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.4': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.5': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.6': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.1': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.2': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.3': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.4': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.5': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.6': {'kdb': True, 'sha1': 'distro'}} 2021-07-10T09:06:46.126 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-07-10T09:06:46.127 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2021-07-10T09:06:46.146 INFO:teuthology.orchestra.run.smithi062.stdout:4.18.0-301.1.el8.x86_64 2021-07-10T09:06:46.147 INFO:teuthology.task.kernel:Running kernel on smithi062: 4.18.0-301.1.el8.x86_64 2021-07-10T09:06:46.147 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum install -y kernel 2021-07-10T09:06:48.756 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-8 - AppStream 50 MB/s | 11 MB 00:00 2021-07-10T09:06:50.868 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-8 - Base 52 MB/s | 8.1 MB 00:00 2021-07-10T09:06:52.203 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-8 - Extras 194 kB/s | 14 kB 00:00 2021-07-10T09:06:52.876 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 17 MB/s | 10 MB 00:00 2021-07-10T09:06:55.108 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-07-10T09:06:56.619 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-07-10T09:06:56.620 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-07-10T09:06:56.620 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-07-10T09:06:56.864 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:06:56.865 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:06:56.865 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2021-07-10T09:06:56.865 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:06:56.866 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:06:56.866 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 4.18.0-315.el8 CentOS-Base 6.4 M 2021-07-10T09:06:56.866 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 4.18.0-315.el8 CentOS-Base 37 M 2021-07-10T09:06:56.866 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 4.18.0-315.el8 CentOS-Base 29 M 2021-07-10T09:06:56.867 INFO:teuthology.orchestra.run.smithi062.stdout:Removing: 2021-07-10T09:06:56.867 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-07-10T09:06:56.867 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-07-10T09:06:56.867 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-07-10T09:06:56.867 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:06:56.868 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:06:56.868 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:06:56.868 INFO:teuthology.orchestra.run.smithi062.stdout:Install 3 Packages 2021-07-10T09:06:56.868 INFO:teuthology.orchestra.run.smithi062.stdout:Remove 3 Packages 2021-07-10T09:06:56.869 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:06:56.869 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 72 M 2021-07-10T09:06:56.870 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:06:57.002 INFO:teuthology.orchestra.run.smithi062.stdout:(1/3): kernel-4.18.0-315.el8.x86_64.rpm 47 MB/s | 6.4 MB 00:00 2021-07-10T09:06:57.747 INFO:teuthology.orchestra.run.smithi062.stdout:(2/3): kernel-modules-4.18.0-315.el8.x86_64.rpm 33 MB/s | 29 MB 00:00 2021-07-10T09:06:58.049 INFO:teuthology.orchestra.run.smithi062.stdout:(3/3): kernel-core-4.18.0-315.el8.x86_64.rpm 31 MB/s | 37 MB 00:01 2021-07-10T09:06:58.054 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:06:58.054 INFO:teuthology.orchestra.run.smithi062.stdout:Total 60 MB/s | 72 MB 00:01 2021-07-10T09:06:58.181 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:06:58.874 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:06:58.874 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:07:00.026 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:07:00.060 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:07:02.805 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:07:03.360 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-core-4.18.0-315.el8.x86_64 1/6 2021-07-10T09:07:07.731 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-315.el8.x86_64 1/6 2021-07-10T09:07:08.078 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-4.18.0-315.el8.x86_64 2/6 2021-07-10T09:07:11.315 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-4.18.0-315.el8.x86_64 2/6 2021-07-10T09:07:11.480 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-4.18.0-315.el8.x86_64 3/6 2021-07-10T09:07:11.481 INFO:teuthology.orchestra.run.smithi062.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-07-10T09:07:11.730 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-07-10T09:07:11.730 INFO:teuthology.orchestra.run.smithi062.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-07-10T09:07:13.983 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-07-10T09:07:14.198 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:07:14.208 INFO:teuthology.orchestra.run.smithi062.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:07:17.326 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:07:39.417 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-315.el8.x86_64 6/6 2021-07-10T09:07:40.021 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:07:40.021 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-4.18.0-315.el8.x86_64 1/6 2021-07-10T09:07:40.021 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-4.18.0-315.el8.x86_64 2/6 2021-07-10T09:07:40.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-4.18.0-315.el8.x86_64 3/6 2021-07-10T09:07:40.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-07-10T09:07:40.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-07-10T09:07:40.454 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:07:40.455 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:07:40.455 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:07:40.455 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-4.18.0-315.el8.x86_64 kernel-core-4.18.0-315.el8.x86_64 2021-07-10T09:07:40.455 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-4.18.0-315.el8.x86_64 2021-07-10T09:07:40.456 INFO:teuthology.orchestra.run.smithi062.stdout:Removed: 2021-07-10T09:07:40.456 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-07-10T09:07:40.456 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-07-10T09:07:40.456 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:07:40.457 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:07:40.560 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel --last | head -n 1 2021-07-10T09:07:40.716 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-4.18.0-315.el8.x86_64 Sat 10 Jul 2021 09:07:11 AM UTC 2021-07-10T09:07:40.718 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-315.el8.x86_64 2021-07-10T09:07:40.719 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-315.el8.x86_64 2021-07-10T09:07:40.719 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2021-07-10T09:07:40.739 INFO:teuthology.orchestra.run.smithi105.stdout:4.18.0-301.1.el8.x86_64 2021-07-10T09:07:40.739 INFO:teuthology.task.kernel:Running kernel on smithi105: 4.18.0-301.1.el8.x86_64 2021-07-10T09:07:40.739 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2021-07-10T09:07:43.416 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-8 - AppStream 52 MB/s | 11 MB 00:00 2021-07-10T09:07:45.631 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-8 - Base 53 MB/s | 8.1 MB 00:00 2021-07-10T09:07:46.995 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-8 - Extras 217 kB/s | 14 kB 00:00 2021-07-10T09:07:47.798 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 14 MB/s | 10 MB 00:00 2021-07-10T09:07:50.050 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-07-10T09:07:51.478 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-07-10T09:07:51.479 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-07-10T09:07:51.479 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-07-10T09:07:51.728 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:07:51.729 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:07:51.729 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2021-07-10T09:07:51.730 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:07:51.730 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:07:51.730 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 4.18.0-315.el8 CentOS-Base 6.4 M 2021-07-10T09:07:51.730 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 4.18.0-315.el8 CentOS-Base 37 M 2021-07-10T09:07:51.731 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 4.18.0-315.el8 CentOS-Base 29 M 2021-07-10T09:07:51.731 INFO:teuthology.orchestra.run.smithi105.stdout:Removing: 2021-07-10T09:07:51.731 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-07-10T09:07:51.731 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-07-10T09:07:51.731 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-07-10T09:07:51.732 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:07:51.732 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:07:51.732 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:07:51.732 INFO:teuthology.orchestra.run.smithi105.stdout:Install 3 Packages 2021-07-10T09:07:51.733 INFO:teuthology.orchestra.run.smithi105.stdout:Remove 3 Packages 2021-07-10T09:07:51.733 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:07:51.733 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 72 M 2021-07-10T09:07:51.734 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:07:52.596 INFO:teuthology.orchestra.run.smithi105.stdout:(1/3): kernel-core-4.18.0-315.el8.x86_64.rpm 42 MB/s | 37 MB 00:00 2021-07-10T09:07:52.615 INFO:teuthology.orchestra.run.smithi105.stdout:(2/3): kernel-4.18.0-315.el8.x86_64.rpm 7.2 MB/s | 6.4 MB 00:00 2021-07-10T09:07:52.914 INFO:teuthology.orchestra.run.smithi105.stdout:(3/3): kernel-modules-4.18.0-315.el8.x86_64.rpm 24 MB/s | 29 MB 00:01 2021-07-10T09:07:52.918 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:07:52.918 INFO:teuthology.orchestra.run.smithi105.stdout:Total 60 MB/s | 72 MB 00:01 2021-07-10T09:07:53.038 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:07:53.685 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:07:53.686 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:07:54.763 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:07:54.775 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:07:57.390 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:07:57.876 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-4.18.0-315.el8.x86_64 1/6 2021-07-10T09:08:02.042 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-4.18.0-315.el8.x86_64 1/6 2021-07-10T09:08:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-4.18.0-315.el8.x86_64 2/6 2021-07-10T09:08:05.424 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-4.18.0-315.el8.x86_64 2/6 2021-07-10T09:08:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-4.18.0-315.el8.x86_64 3/6 2021-07-10T09:08:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-07-10T09:08:05.790 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-07-10T09:08:05.790 INFO:teuthology.orchestra.run.smithi105.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-07-10T09:08:07.974 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-07-10T09:08:08.166 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:08:08.177 INFO:teuthology.orchestra.run.smithi105.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:08:11.014 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:08:31.406 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-4.18.0-315.el8.x86_64 6/6 2021-07-10T09:08:31.766 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:08:31.766 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-4.18.0-315.el8.x86_64 1/6 2021-07-10T09:08:31.766 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-4.18.0-315.el8.x86_64 2/6 2021-07-10T09:08:31.767 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-4.18.0-315.el8.x86_64 3/6 2021-07-10T09:08:31.767 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-07-10T09:08:31.767 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-07-10T09:08:32.240 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-07-10T09:08:32.241 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:08:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:08:32.241 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-4.18.0-315.el8.x86_64 kernel-core-4.18.0-315.el8.x86_64 2021-07-10T09:08:32.241 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-4.18.0-315.el8.x86_64 2021-07-10T09:08:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:Removed: 2021-07-10T09:08:32.242 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-07-10T09:08:32.242 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-07-10T09:08:32.242 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:08:32.243 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:08:32.344 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel --last | head -n 1 2021-07-10T09:08:32.501 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-4.18.0-315.el8.x86_64 Sat 10 Jul 2021 09:08:05 AM UTC 2021-07-10T09:08:32.502 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-315.el8.x86_64 2021-07-10T09:08:32.503 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-315.el8.x86_64 2021-07-10T09:08:32.503 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-07-10T09:08:32.504 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-07-10T09:08:32.504 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-07-10T09:08:32.504 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2021-07-10T09:08:32.505 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2021-07-10T09:08:32.505 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel --last | head -n 1 2021-07-10T09:08:32.660 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-4.18.0-315.el8.x86_64 Sat 10 Jul 2021 09:07:11 AM UTC 2021-07-10T09:08:32.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-315.el8.x86_64 2021-07-10T09:08:32.662 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2021-07-10T09:08:32.695 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2021-07-10T09:08:32.696 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2021-07-10T09:08:32.696 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.02 2021-07-10T09:08:32.696 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 99.el8 2021-07-10T09:08:32.697 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2021-07-10T09:08:32.697 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-07-10T09:08:32.698 INFO:teuthology.orchestra.run.smithi062.stdout:Group : System Environment/Base 2021-07-10T09:08:32.698 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 9264704 2021-07-10T09:08:32.699 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2021-07-10T09:08:32.699 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-07-10T09:08:32.699 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-07-10T09:08:32.700 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-07-10T09:08:32.700 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : kbuilder.bsys.centos.org 2021-07-10T09:08:32.701 INFO:teuthology.orchestra.run.smithi062.stdout:Relocations : (not relocatable) 2021-07-10T09:08:32.701 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : CentOS BuildSystem 2021-07-10T09:08:32.702 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2021-07-10T09:08:32.702 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2021-07-10T09:08:32.703 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2021-07-10T09:08:32.703 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2021-07-10T09:08:32.703 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:08:32.704 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-07-10T09:08:32.704 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2021-07-10T09:08:32.705 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2021-07-10T09:08:32.705 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2021-07-10T09:08:32.706 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:08:32.706 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2021-07-10T09:08:32.708 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-07-10T09:08:32.709 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 4.18.0-315.el8.x86_64 2021-07-10T09:08:32.710 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-07-10T09:08:33.446 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2021-07-10T09:08:35.828 INFO:teuthology.orchestra.run.smithi062.stderr:done 2021-07-10T09:08:35.830 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2021-07-10T09:08:35.870 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.mMdu600YjI 2021-07-10T09:08:35.871 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mMdu600YjI 2021-07-10T09:08:35.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.mMdu600YjI 2021-07-10T09:08:36.108 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.mMdu600YjI is 6KB 2021-07-10T09:08:36.158 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.mMdu600YjI 2021-07-10T09:08:36.175 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2021-07-10T09:08:36.238 INFO:teuthology.orchestra.run.smithi062.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-07-10T09:08:36.238 INFO:teuthology.orchestra.run.smithi062.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-07-10T09:08:36.239 INFO:teuthology.orchestra.run.smithi062.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-07-10T09:08:36.239 INFO:teuthology.orchestra.run.smithi062.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-315.el8.x86_64.conf 2021-07-10T09:08:36.240 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-315.el8.x86_64 2021-07-10T09:08:36.412 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2021-07-10T09:08:36.414 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2021-07-10T09:08:36.414 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2021-07-10T09:08:36.415 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel --last | head -n 1 2021-07-10T09:08:36.560 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-4.18.0-315.el8.x86_64 Sat 10 Jul 2021 09:08:05 AM UTC 2021-07-10T09:08:36.561 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-315.el8.x86_64 2021-07-10T09:08:36.562 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2021-07-10T09:08:36.592 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2021-07-10T09:08:36.592 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2021-07-10T09:08:36.593 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.02 2021-07-10T09:08:36.593 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 99.el8 2021-07-10T09:08:36.593 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2021-07-10T09:08:36.593 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-07-10T09:08:36.593 INFO:teuthology.orchestra.run.smithi105.stdout:Group : System Environment/Base 2021-07-10T09:08:36.594 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 9264704 2021-07-10T09:08:36.594 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2021-07-10T09:08:36.594 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-07-10T09:08:36.594 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-07-10T09:08:36.595 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-07-10T09:08:36.595 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : kbuilder.bsys.centos.org 2021-07-10T09:08:36.595 INFO:teuthology.orchestra.run.smithi105.stdout:Relocations : (not relocatable) 2021-07-10T09:08:36.595 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : CentOS BuildSystem 2021-07-10T09:08:36.596 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2021-07-10T09:08:36.596 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2021-07-10T09:08:36.596 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2021-07-10T09:08:36.596 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2021-07-10T09:08:36.596 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:08:36.597 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-07-10T09:08:36.597 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2021-07-10T09:08:36.597 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2021-07-10T09:08:36.597 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2021-07-10T09:08:36.597 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:08:36.598 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2021-07-10T09:08:36.599 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-07-10T09:08:36.599 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 4.18.0-315.el8.x86_64 2021-07-10T09:08:36.599 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-07-10T09:08:37.738 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2021-07-10T09:08:39.629 INFO:teuthology.orchestra.run.smithi105.stderr:done 2021-07-10T09:08:39.631 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2021-07-10T09:08:39.669 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.4gTL2OYk4G 2021-07-10T09:08:39.669 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4gTL2OYk4G 2021-07-10T09:08:39.743 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.4gTL2OYk4G 2021-07-10T09:08:39.877 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.4gTL2OYk4G is 6KB 2021-07-10T09:08:39.926 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.4gTL2OYk4G 2021-07-10T09:08:39.941 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2021-07-10T09:08:40.003 INFO:teuthology.orchestra.run.smithi105.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-07-10T09:08:40.003 INFO:teuthology.orchestra.run.smithi105.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-07-10T09:08:40.004 INFO:teuthology.orchestra.run.smithi105.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-07-10T09:08:40.004 INFO:teuthology.orchestra.run.smithi105.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-315.el8.x86_64.conf 2021-07-10T09:08:40.005 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-315.el8.x86_64 2021-07-10T09:08:40.187 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2021-07-10T09:09:10.219 INFO:teuthology.misc:Re-opening connections... 2021-07-10T09:09:10.219 INFO:teuthology.misc:trying to connect to ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:09:10.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-10T09:09:10.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:09:28.651 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2021-07-10T09:09:28.652 INFO:teuthology.misc:trying to connect to ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:09:28.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-10T09:09:28.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:10:28.715 DEBUG:teuthology.orchestra.remote:timed out 2021-07-10T09:10:28.717 DEBUG:teuthology.misc:waited 78.49748063087463 2021-07-10T09:10:29.718 INFO:teuthology.misc:trying to connect to ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:10:29.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-10T09:10:29.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:10:29.998 DEBUG:teuthology.orchestra.run.smithi062:> true 2021-07-10T09:10:30.262 DEBUG:teuthology.misc:waited 80.04282307624817 2021-07-10T09:10:31.263 INFO:teuthology.misc:trying to connect to ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:10:31.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-10T09:10:31.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:10:31.512 DEBUG:teuthology.orchestra.run.smithi105:> true 2021-07-10T09:10:31.779 DEBUG:teuthology.misc:waited 81.55989956855774 2021-07-10T09:10:32.781 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2021-07-10T09:10:32.781 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "4.18.0-315.el8.x86_64"... 2021-07-10T09:10:32.782 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2021-07-10T09:10:32.801 INFO:teuthology.orchestra.run.smithi062.stdout:4.18.0-315.el8.x86_64 2021-07-10T09:10:32.802 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-315.el8.x86_64 vs 4.18.0-315.el8.x86_64 2021-07-10T09:10:32.802 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-07-10T09:10:32.802 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-07-10T09:10:32.803 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2021-07-10T09:10:32.803 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "4.18.0-315.el8.x86_64"... 2021-07-10T09:10:32.803 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2021-07-10T09:10:32.822 INFO:teuthology.orchestra.run.smithi105.stdout:4.18.0-315.el8.x86_64 2021-07-10T09:10:32.823 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-315.el8.x86_64 vs 4.18.0-315.el8.x86_64 2021-07-10T09:10:32.823 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-07-10T09:10:32.823 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-07-10T09:10:33.825 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2021-07-10T09:10:33.833 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-07-10T09:10:33.916 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2021-07-10T09:10:33.929 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2021-07-10T09:10:33.930 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-07-10T09:10:34.023 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2021-07-10T09:10:34.035 INFO:teuthology.run_tasks:Running task internal.base... 2021-07-10T09:10:34.044 INFO:teuthology.task.internal:Creating test directory... 2021-07-10T09:10:34.045 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-07-10T09:10:34.047 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-07-10T09:10:34.065 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-07-10T09:10:34.077 INFO:teuthology.run_tasks:Running task internal.archive... 2021-07-10T09:10:34.116 INFO:teuthology.task.internal:Creating archive directory... 2021-07-10T09:10:34.117 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-07-10T09:10:34.122 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-07-10T09:10:34.158 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-07-10T09:10:34.170 INFO:teuthology.task.internal:Enabling coredump saving... 2021-07-10T09:10:34.170 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-07-10T09:10:34.186 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-07-10T09:10:34.217 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-10T09:10:34.218 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-10T09:10:34.229 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-10T09:10:34.230 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-10T09:10:34.231 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-07-10T09:10:34.246 INFO:teuthology.task.internal:Configuring sudo... 2021-07-10T09:10:34.247 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-07-10T09:10:34.273 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-07-10T09:10:34.303 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-07-10T09:10:34.319 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-07-10T09:10:34.319 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-07-10T09:10:34.345 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-07-10T09:10:34.363 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-10T09:10:34.438 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-10T09:10:34.508 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-10T09:10:34.542 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-10T09:10:34.593 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:10:34.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-07-10T09:10:34.660 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-10T09:10:34.699 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-10T09:10:34.757 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-10T09:10:34.835 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-10T09:10:34.884 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:10:34.885 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-07-10T09:10:34.953 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2021-07-10T09:10:34.955 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2021-07-10T09:10:35.012 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-07-10T09:10:35.050 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-07-10T09:10:35.490 INFO:teuthology.run_tasks:Running task internal.timer... 2021-07-10T09:10:35.500 INFO:teuthology.task.internal:Starting timer... 2021-07-10T09:10:35.501 INFO:teuthology.run_tasks:Running task pcp... 2021-07-10T09:10:35.541 INFO:teuthology.run_tasks:Running task selinux... 2021-07-10T09:10:35.573 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-07-10T09:10:35.574 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2021-07-10T09:10:35.635 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: [ OK ] 2021-07-10T09:10:35.636 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2021-07-10T09:10:35.689 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: [ OK ] 2021-07-10T09:10:35.690 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-07-10T09:10:35.690 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2021-07-10T09:10:35.716 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2021-07-10T09:10:35.717 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2021-07-10T09:10:35.735 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2021-07-10T09:10:35.735 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive'} 2021-07-10T09:10:35.736 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-07-10T09:10:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:10:35.785 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-07-10T09:10:35.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:10:35.810 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-07-10T09:10:35.810 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2021-07-10T09:10:35.861 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2021-07-10T09:10:35.902 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-07-10T09:10:35.913 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-07-10T09:10:35.920 INFO:teuthology.repo_utils:Fetching master from origin 2021-07-10T09:10:35.956 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-07-10T09:10:35.979 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-07-10T09:10:35.982 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-07-10T09:10:36.363 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-07-10T09:10:37.212 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-07-10T09:10:40.217 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-07-10T09:10:40.240 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:40 +0000 (0:00:00.040) 0:00:00.040 ********* 2021-07-10T09:10:40.251 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:40 +0000 (0:00:00.011) 0:00:00.051 ********* 2021-07-10T09:10:41.743 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi105.front.sepia.ceph.com] 2021-07-10T09:10:41.790 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:41 +0000 (0:00:01.538) 0:00:01.590 ********* 2021-07-10T09:10:42.244 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-07-10T09:10:42.246 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-10T09:10:42.266 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:42 +0000 (0:00:00.476) 0:00:02.066 ********* 2021-07-10T09:10:42.788 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi062.front.sepia.ceph.com] 2021-07-10T09:10:42.836 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:42 +0000 (0:00:00.569) 0:00:02.636 ********* 2021-07-10T09:10:42.874 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-07-10T09:10:42.875 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-07-10T09:10:43.046 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:43 +0000 (0:00:00.210) 0:00:02.846 ********* 2021-07-10T09:10:43.057 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-10T09:10:43.091 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:43 +0000 (0:00:00.044) 0:00:02.890 ********* 2021-07-10T09:10:43.164 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:43.194 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:43 +0000 (0:00:00.103) 0:00:02.994 ********* 2021-07-10T09:10:43.437 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-07-10T09:10:43.439 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-07-10T09:10:43.461 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:43 +0000 (0:00:00.266) 0:00:03.261 ********* 2021-07-10T09:10:43.552 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-07-10T09:10:43.554 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:43.575 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:43 +0000 (0:00:00.114) 0:00:03.375 ********* 2021-07-10T09:10:43.582 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-07-10T09:10:43.584 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:10:43.609 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:43 +0000 (0:00:00.033) 0:00:03.409 ********* 2021-07-10T09:10:43.786 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:10:43.816 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:43 +0000 (0:00:00.206) 0:00:03.616 ********* 2021-07-10T09:10:44.680 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-07-10T09:10:44.682 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908243.8784904-29451-170700324674866/source", "state": "file", "uid": 0} changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908243.6659453-29439-143810843619728/source", "state": "file", "uid": 0} 2021-07-10T09:10:44.704 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:44 +0000 (0:00:00.888) 0:00:04.504 ********* 2021-07-10T09:10:44.726 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:44 +0000 (0:00:00.021) 0:00:04.526 ********* 2021-07-10T09:10:45.164 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-07-10T09:10:45.166 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-07-10T09:10:45.168 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-07-10T09:10:45.190 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-07-10T09:10:45.192 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-07-10T09:10:45.264 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-07-10T09:10:45.267 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-10T09:10:45.279 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 10 July 2021 09:10:45 +0000 (0:00:00.553) 0:00:05.080 ********* 2021-07-10T09:10:45.814 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] 2021-07-10T09:10:46.820 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] 2021-07-10T09:10:46.837 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-07-10T09:10:46.839 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:46 +0000 (0:00:01.557) 0:00:06.638 ********* 2021-07-10T09:10:46.865 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-07-10T09:10:46.876 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-07-10T09:10:46.881 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-10T09:10:47.031 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:47 +0000 (0:00:00.193) 0:00:06.831 ********* 2021-07-10T09:10:47.053 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:47 +0000 (0:00:00.021) 0:00:06.853 ********* 2021-07-10T09:10:47.703 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi105.front.sepia.ceph.com] 2021-07-10T09:10:47.713 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] 2021-07-10T09:10:47.777 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Saturday 10 July 2021 09:10:47 +0000 (0:00:00.723) 0:00:07.577 ********* 2021-07-10T09:10:47.805 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:47 +0000 (0:00:00.026) 0:00:07.603 ********* 2021-07-10T09:10:48.335 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-07-10T09:10:48.337 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.183327", "end": "2021-07-10 09:10:48.309159", "rc": 0, "start": "2021-07-10 09:10:48.125832", "stderr": "", "stderr_lines": [], "stdout": "444af9608e8be25a2d7a2892fb588d0daeff9780", "stdout_lines": ["444af9608e8be25a2d7a2892fb588d0daeff9780"]} 2021-07-10T09:10:48.359 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.555) 0:00:08.159 ********* 2021-07-10T09:10:48.368 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.197860", "end": "2021-07-10 09:10:48.338499", "rc": 0, "start": "2021-07-10 09:10:48.140639", "stderr": "", "stderr_lines": [], "stdout": "444af9608e8be25a2d7a2892fb588d0daeff9780", "stdout_lines": ["444af9608e8be25a2d7a2892fb588d0daeff9780"]} 2021-07-10T09:10:48.393 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.033) 0:00:08.193 ********* 2021-07-10T09:10:48.576 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-07-10T09:10:48.578 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014120", "end": "2021-07-10 09:10:48.551343", "failed_when_result": false, "rc": 0, "start": "2021-07-10 09:10:48.537223", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-07-10T09:10:48.598 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.205) 0:00:08.398 ********* 2021-07-10T09:10:48.607 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010167", "end": "2021-07-10 09:10:48.567627", "failed_when_result": false, "rc": 0, "start": "2021-07-10 09:10:48.557460", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-07-10T09:10:48.635 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.036) 0:00:08.435 ********* 2021-07-10T09:10:48.675 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-07-10T09:10:48.677 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:48.698 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.063) 0:00:08.498 ********* 2021-07-10T09:10:48.715 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:48.738 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.040) 0:00:08.538 ********* 2021-07-10T09:10:48.768 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:48.791 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.052) 0:00:08.591 ********* 2021-07-10T09:10:48.809 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:48.833 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.041) 0:00:08.633 ********* 2021-07-10T09:10:48.865 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-07-10T09:10:48.866 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:48.889 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.055) 0:00:08.689 ********* 2021-07-10T09:10:48.909 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:48.944 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.054) 0:00:08.744 ********* 2021-07-10T09:10:48.965 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-07-10T09:10:48.967 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:48.989 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:48 +0000 (0:00:00.044) 0:00:08.789 ********* 2021-07-10T09:10:49.016 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:49.040 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:49 +0000 (0:00:00.051) 0:00:08.840 ********* 2021-07-10T09:10:49.070 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:49.093 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:49 +0000 (0:00:00.052) 0:00:08.893 ********* 2021-07-10T09:10:49.111 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:10:49.135 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:49 +0000 (0:00:00.041) 0:00:08.935 ********* 2021-07-10T09:10:49.352 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-07-10T09:10:49.354 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:10:49.376 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:49 +0000 (0:00:00.241) 0:00:09.176 ********* 2021-07-10T09:10:49.384 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:10:49.409 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:10:49 +0000 (0:00:00.033) 0:00:09.209 ********* 2021-07-10T09:10:49.677 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-10T09:10:49.679 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-07-10T09:10:49.710 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-07-10T09:10:49.878 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-07-10T09:10:49.901 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-07-10T09:10:50.089 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-07-10T09:10:50.101 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-07-10T09:10:50.300 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-07-10T09:10:50.312 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-07-10T09:10:50.501 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-07-10T09:10:50.513 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-07-10T09:10:50.712 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-07-10T09:10:50.714 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-07-10T09:10:50.904 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-07-10T09:10:50.916 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-07-10T09:10:51.111 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-07-10T09:10:51.122 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-07-10T09:10:51.312 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-07-10T09:10:51.334 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-07-10T09:10:51.513 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-07-10T09:10:51.546 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-07-10T09:10:51.713 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-07-10T09:10:51.747 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-07-10T09:10:51.903 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-07-10T09:10:51.958 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-07-10T09:10:52.106 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-07-10T09:10:52.182 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-07-10T09:10:52.298 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-07-10T09:10:52.394 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-07-10T09:10:52.500 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-07-10T09:10:52.595 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-07-10T09:10:52.701 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-07-10T09:10:52.806 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-07-10T09:10:52.891 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-07-10T09:10:53.007 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-07-10T09:10:53.092 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-07-10T09:10:53.217 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-07-10T09:10:53.281 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-07-10T09:10:53.418 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-07-10T09:10:53.481 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-07-10T09:10:53.627 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-07-10T09:10:53.681 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-07-10T09:10:53.828 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-07-10T09:10:53.871 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-07-10T09:10:54.043 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-07-10T09:10:54.076 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-07-10T09:10:54.249 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-07-10T09:10:54.273 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-07-10T09:10:54.453 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-07-10T09:10:54.465 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-07-10T09:10:54.656 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-07-10T09:10:54.668 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-07-10T09:10:54.861 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-07-10T09:10:54.876 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-07-10T09:10:55.063 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-07-10T09:10:55.077 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-07-10T09:10:55.256 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-07-10T09:10:55.290 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-07-10T09:10:55.460 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-07-10T09:10:55.503 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-07-10T09:10:55.651 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-07-10T09:10:55.705 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-07-10T09:10:55.844 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-07-10T09:10:55.909 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-07-10T09:10:56.047 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-07-10T09:10:56.121 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-07-10T09:10:56.234 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-07-10T09:10:56.328 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-07-10T09:10:56.434 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-07-10T09:10:56.530 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-07-10T09:10:56.626 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-07-10T09:10:56.733 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-07-10T09:10:56.828 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-07-10T09:10:56.944 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-07-10T09:10:57.019 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-07-10T09:10:57.147 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-07-10T09:10:57.221 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-10T09:10:57.358 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-07-10T09:10:57.422 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-10T09:10:57.560 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-10T09:10:57.614 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-07-10T09:10:57.773 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-10T09:10:57.807 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-07-10T09:10:57.987 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-07-10T09:10:58.009 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-07-10T09:10:58.199 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-07-10T09:10:58.210 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-07-10T09:10:58.410 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-07-10T09:10:58.411 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-07-10T09:10:58.609 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-10T09:10:58.621 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-07-10T09:10:58.809 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-10T09:10:58.821 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-07-10T09:10:59.000 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-10T09:10:59.033 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-10T09:10:59.204 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-07-10T09:10:59.238 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-10T09:10:59.395 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-07-10T09:10:59.449 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-10T09:10:59.600 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-07-10T09:10:59.654 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-07-10T09:10:59.804 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-07-10T09:10:59.869 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-07-10T09:11:00.006 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-07-10T09:11:00.082 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-07-10T09:11:00.199 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-07-10T09:11:00.294 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-07-10T09:11:00.400 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-07-10T09:11:00.497 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-07-10T09:11:00.593 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-07-10T09:11:00.709 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-07-10T09:11:00.796 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-07-10T09:11:00.912 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-07-10T09:11:00.997 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-07-10T09:11:01.125 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-07-10T09:11:01.199 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-07-10T09:11:01.335 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-07-10T09:11:01.400 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-07-10T09:11:01.548 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-07-10T09:11:01.600 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-07-10T09:11:01.756 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-07-10T09:11:01.798 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-07-10T09:11:01.976 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-07-10T09:11:01.998 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-07-10T09:11:02.177 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-07-10T09:11:02.198 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-07-10T09:11:02.380 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-07-10T09:11:02.393 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-07-10T09:11:02.587 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-07-10T09:11:02.598 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-07-10T09:11:02.798 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-07-10T09:11:02.799 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-07-10T09:11:02.999 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-07-10T09:11:03.000 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-07-10T09:11:03.200 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-07-10T09:11:03.212 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-07-10T09:11:03.393 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-07-10T09:11:03.415 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-07-10T09:11:03.584 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-07-10T09:11:03.627 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-07-10T09:11:03.788 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-07-10T09:11:03.831 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-07-10T09:11:04.019 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-07-10T09:11:04.055 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-07-10T09:11:04.225 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-07-10T09:11:04.276 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-07-10T09:11:04.425 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-07-10T09:11:04.490 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-07-10T09:11:04.618 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-07-10T09:11:04.691 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-07-10T09:11:04.828 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-07-10T09:11:04.903 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-07-10T09:11:05.029 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-07-10T09:11:05.115 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-07-10T09:11:05.209 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-07-10T09:11:05.316 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-07-10T09:11:05.411 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-07-10T09:11:05.518 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-07-10T09:11:05.624 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-07-10T09:11:05.731 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-07-10T09:11:05.816 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-07-10T09:11:05.933 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-07-10T09:11:06.019 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-07-10T09:11:06.145 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-07-10T09:11:06.209 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-07-10T09:11:06.347 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-07-10T09:11:06.412 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-07-10T09:11:06.560 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-07-10T09:11:06.603 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-07-10T09:11:06.770 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-07-10T09:11:06.802 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-07-10T09:11:06.971 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-07-10T09:11:07.004 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-07-10T09:11:07.186 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-07-10T09:11:07.208 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-07-10T09:11:07.402 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-07-10T09:11:07.415 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-07-10T09:11:07.604 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-07-10T09:11:07.616 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-07-10T09:11:07.805 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-07-10T09:11:07.816 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-07-10T09:11:08.006 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-07-10T09:11:08.018 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-07-10T09:11:08.198 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-07-10T09:11:08.231 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-07-10T09:11:08.402 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-07-10T09:11:08.435 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-07-10T09:11:08.603 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-07-10T09:11:08.645 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-07-10T09:11:08.794 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-07-10T09:11:08.858 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-07-10T09:11:09.000 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-07-10T09:11:09.068 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-07-10T09:11:09.217 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-07-10T09:11:09.303 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-07-10T09:11:09.471 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-07-10T09:11:09.531 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-07-10T09:11:09.673 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-10T09:11:09.780 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-07-10T09:11:09.887 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-07-10T09:11:10.016 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-07-10T09:11:10.092 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-07-10T09:11:10.280 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-07-10T09:11:10.314 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-07-10T09:11:10.505 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-07-10T09:11:10.516 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-07-10T09:11:10.717 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-07-10T09:11:10.729 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-10T09:11:10.919 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-07-10T09:11:10.929 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-07-10T09:11:11.119 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-07-10T09:11:11.132 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-07-10T09:11:11.314 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-07-10T09:11:11.347 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-07-10T09:11:11.516 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-07-10T09:11:11.560 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-07-10T09:11:11.707 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-07-10T09:11:11.773 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-07-10T09:11:11.910 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-07-10T09:11:11.964 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-07-10T09:11:12.100 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-07-10T09:11:12.174 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-07-10T09:11:12.300 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-07-10T09:11:12.374 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-07-10T09:11:12.500 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-07-10T09:11:12.585 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-07-10T09:11:12.702 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-07-10T09:11:12.797 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-07-10T09:11:12.902 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-07-10T09:11:13.007 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-07-10T09:11:13.102 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-07-10T09:11:13.211 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-07-10T09:11:13.292 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-07-10T09:11:13.418 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-07-10T09:11:13.493 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-07-10T09:11:13.619 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-07-10T09:11:13.683 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-07-10T09:11:13.831 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-07-10T09:11:13.895 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-07-10T09:11:14.031 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-07-10T09:11:14.095 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-07-10T09:11:14.245 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-07-10T09:11:14.287 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-07-10T09:11:14.448 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-07-10T09:11:14.480 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-07-10T09:11:14.649 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-07-10T09:11:14.682 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-07-10T09:11:14.871 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-07-10T09:11:14.883 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-07-10T09:11:15.062 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-07-10T09:11:15.074 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-07-10T09:11:15.274 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-07-10T09:11:15.275 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-07-10T09:11:15.477 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-07-10T09:11:15.479 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-07-10T09:11:15.670 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-07-10T09:11:15.692 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-07-10T09:11:15.870 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-07-10T09:11:15.892 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-07-10T09:11:16.070 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-07-10T09:11:16.103 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-07-10T09:11:16.260 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-07-10T09:11:16.323 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-07-10T09:11:16.465 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-07-10T09:11:16.529 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-07-10T09:11:16.665 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-07-10T09:11:16.728 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-07-10T09:11:16.855 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-07-10T09:11:16.940 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-07-10T09:11:17.057 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-07-10T09:11:17.141 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-07-10T09:11:17.247 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-07-10T09:11:17.351 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-07-10T09:11:17.560 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-07-10T09:11:17.770 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-07-10T09:11:17.970 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-07-10T09:11:18.180 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-07-10T09:11:18.389 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-07-10T09:11:18.598 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-07-10T09:11:18.795 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-07-10T09:11:20.233 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-07-10T09:11:20.541 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-07-10T09:11:22.016 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-07-10T09:11:22.080 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-07-10T09:11:23.555 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-07-10T09:11:23.619 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-07-10T09:11:25.133 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-07-10T09:11:25.238 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-07-10T09:11:26.615 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-07-10T09:11:26.816 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-07-10T09:11:28.167 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-07-10T09:11:28.459 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-07-10T09:11:29.705 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-07-10T09:11:30.275 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-07-10T09:11:31.292 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-07-10T09:11:31.845 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-07-10T09:11:32.799 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-07-10T09:11:33.437 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-07-10T09:11:34.333 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-07-10T09:11:35.071 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-07-10T09:11:35.878 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-07-10T09:11:36.750 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-07-10T09:11:37.521 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-07-10T09:11:38.454 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-07-10T09:11:39.112 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-07-10T09:11:39.251 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:39 +0000 (0:00:49.841) 0:00:59.051 ********* 2021-07-10T09:11:39.319 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-07-10T09:11:39.340 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:39 +0000 (0:00:00.089) 0:00:59.141 ********* 2021-07-10T09:11:39.432 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-07-10T09:11:39.434 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-07-10T09:11:39.435 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-07-10T09:11:39.437 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-07-10T09:11:39.439 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-07-10T09:11:39.440 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1 2021-07-10T09:11:39.441 INFO:teuthology.task.ansible.out:eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-07-10T09:11:39.443 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-07-10T09:11:39.444 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-07-10T09:11:39.446 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-07-10T09:11:39.447 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-07-10T09:11:39.449 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-07-10T09:11:39.450 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-07-10T09:11:39.452 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0 2021-07-10T09:11:39.453 INFO:teuthology.task.ansible.out:fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-07-10T09:11:39.454 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-07-10T09:11:39.456 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}]}, "changed": false} 2021-07-10T09:11:39.479 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:39 +0000 (0:00:00.138) 0:00:59.279 ********* 2021-07-10T09:11:40.176 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-10T09:11:40.178 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-07-10T09:11:40.419 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-07-10T09:11:40.421 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"after": "444af9608e8be25a2d7a2892fb588d0daeff9780", "attempts": 1, "before": "444af9608e8be25a2d7a2892fb588d0daeff9780", "changed": false, "remote_url_changed": false} 2021-07-10T09:11:40.441 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:40 +0000 (0:00:00.961) 0:01:00.241 ********* 2021-07-10T09:11:40.763 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-07-10T09:11:40.970 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-07-10T09:11:41.179 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-07-10T09:11:41.389 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-07-10T09:11:41.391 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-07-10T09:11:41.609 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-07-10T09:11:41.818 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-07-10T09:11:41.820 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-07-10T09:11:41.840 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-10T09:11:41.841 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-07-10T09:11:41.971 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:41 +0000 (0:00:01.529) 0:01:01.771 ********* 2021-07-10T09:11:42.031 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:11:42.033 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-07-10T09:11:42.043 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-07-10T09:11:42.064 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:42 +0000 (0:00:00.093) 0:01:01.864 ********* 2021-07-10T09:11:42.158 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c 2021-07-10T09:11:42.160 INFO:teuthology.task.ansible.out:68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "a 2021-07-10T09:11:42.162 INFO:teuthology.task.ansible.out:li@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb27 2021-07-10T09:11:42.163 INFO:teuthology.task.ansible.out:3afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767 2021-07-10T09:11:42.165 INFO:teuthology.task.ansible.out:dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name" 2021-07-10T09:11:42.166 INFO:teuthology.task.ansible.out:: "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42 2021-07-10T09:11:42.168 INFO:teuthology.task.ansible.out:c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3 2021-07-10T09:11:42.170 INFO:teuthology.task.ansible.out:ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf 2021-07-10T09:11:42.172 INFO:teuthology.task.ansible.out:2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc99 2021-07-10T09:11:42.173 INFO:teuthology.task.ansible.out:3327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3 2021-07-10T09:11:42.174 INFO:teuthology.task.ansible.out:f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQ 2021-07-10T09:11:42.176 INFO:teuthology.task.ansible.out:PrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "pr 2021-07-10T09:11:42.178 INFO:teuthology.task.ansible.out:ateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf 2021-07-10T09:11:42.179 INFO:teuthology.task.ansible.out:8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": " 2021-07-10T09:11:42.181 INFO:teuthology.task.ansible.out:cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}]}, "changed": false} 2021-07-10T09:11:42.199 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:42 +0000 (0:00:00.134) 0:01:01.999 ********* 2021-07-10T09:11:42.237 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:11:42.239 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-07-10T09:11:42.438 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-07-10T09:11:42.451 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:11:42.653 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-07-10T09:11:42.654 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-07-10T09:11:42.874 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-07-10T09:11:42.876 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"after": "444af9608e8be25a2d7a2892fb588d0daeff9780", "attempts": 1, "before": "444af9608e8be25a2d7a2892fb588d0daeff9780", "changed": false, "remote_url_changed": false} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KP 2021-07-10T09:11:42.878 INFO:teuthology.task.ansible.out:FZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2021-07-10T09:11:42.898 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:11:42 +0000 (0:00:00.699) 0:01:02.698 ********* 2021-07-10T09:11:42.906 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:11:42.908 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:11:43.107 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-07-10T09:11:43.256 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-07-10T09:11:43.330 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-07-10T09:11:43.332 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-07-10T09:11:43.467 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-07-10T09:11:43.542 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-07-10T09:11:43.689 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-07-10T09:11:43.753 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-07-10T09:11:43.755 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-07-10T09:11:43.919 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-07-10T09:11:43.921 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-07-10T09:11:43.962 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-07-10T09:11:44.142 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-07-10T09:11:44.176 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-07-10T09:11:44.364 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-07-10T09:11:44.366 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-07-10T09:11:44.386 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-07-10T09:11:44.575 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-07-10T09:11:44.607 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-07-10T09:11:44.609 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-07-10T09:11:44.797 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-07-10T09:11:44.808 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-07-10T09:11:44.810 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-07-10T09:11:45.019 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-07-10T09:11:45.031 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:11:45.032 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-07-10T09:11:45.232 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-07-10T09:11:45.243 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-07-10T09:11:45.245 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-07-10T09:11:45.444 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-07-10T09:11:45.446 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-07-10T09:11:45.466 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-07-10T09:11:45.467 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-07-10T09:11:45.477 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:11:45.479 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:11:45.646 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-07-10T09:11:45.710 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-07-10T09:11:45.857 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-07-10T09:11:45.932 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-07-10T09:11:45.933 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-07-10T09:11:46.058 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-07-10T09:11:46.141 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-07-10T09:11:46.269 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-07-10T09:11:46.366 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-07-10T09:11:46.368 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-07-10T09:11:46.471 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-07-10T09:11:46.577 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-07-10T09:11:46.682 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-07-10T09:11:46.684 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-07-10T09:11:46.787 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-07-10T09:11:46.884 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-07-10T09:11:47.010 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-07-10T09:11:47.094 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-07-10T09:11:47.241 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-07-10T09:11:47.243 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-07-10T09:11:47.305 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-07-10T09:11:47.451 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-07-10T09:11:47.453 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-07-10T09:11:47.504 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-07-10T09:11:47.670 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-07-10T09:11:47.712 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-07-10T09:11:47.881 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-07-10T09:11:47.924 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-10T09:11:48.103 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-07-10T09:11:48.104 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-07-10T09:11:48.125 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-07-10T09:11:48.315 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-07-10T09:11:48.337 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-07-10T09:11:48.527 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-07-10T09:11:48.539 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-07-10T09:11:48.541 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-07-10T09:11:48.750 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-07-10T09:11:48.762 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-07-10T09:11:48.764 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-07-10T09:11:48.967 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-07-10T09:11:48.969 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-07-10T09:11:49.175 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-07-10T09:11:49.197 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-07-10T09:11:49.386 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-07-10T09:11:49.407 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-07-10T09:11:49.409 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-07-10T09:11:49.608 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-07-10T09:11:49.609 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-10T09:11:49.629 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-07-10T09:11:49.809 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-07-10T09:11:49.810 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-07-10T09:11:49.812 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-07-10T09:11:49.853 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-07-10T09:11:50.022 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-07-10T09:11:50.066 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-07-10T09:11:50.224 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-07-10T09:11:50.277 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-07-10T09:11:50.425 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-07-10T09:11:50.499 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-07-10T09:11:50.643 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-07-10T09:11:50.719 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-10T09:11:50.845 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-07-10T09:11:50.930 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-07-10T09:11:51.059 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-07-10T09:11:51.060 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-07-10T09:11:51.142 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-07-10T09:11:51.259 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-07-10T09:11:51.364 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-07-10T09:11:51.366 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-07-10T09:11:51.470 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-07-10T09:11:51.585 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-07-10T09:11:51.587 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-07-10T09:11:51.722 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-07-10T09:11:51.724 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-07-10T09:11:51.796 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-07-10T09:11:51.933 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-07-10T09:11:52.017 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-07-10T09:11:52.144 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-07-10T09:11:52.229 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-07-10T09:11:52.356 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-07-10T09:11:52.358 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-07-10T09:11:52.450 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-07-10T09:11:52.452 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-10T09:11:52.565 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-07-10T09:11:52.677 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-07-10T09:11:52.679 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-07-10T09:11:52.681 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-07-10T09:11:52.772 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-07-10T09:11:52.899 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-07-10T09:11:52.984 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-07-10T09:11:53.120 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-07-10T09:11:53.184 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-10T09:11:53.332 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-07-10T09:11:53.396 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-07-10T09:11:53.553 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-07-10T09:11:53.606 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-07-10T09:11:53.763 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-07-10T09:11:53.805 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-07-10T09:11:53.986 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-07-10T09:11:53.988 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-07-10T09:11:54.018 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-07-10T09:11:54.208 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-07-10T09:11:54.219 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-07-10T09:11:54.417 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-07-10T09:11:54.429 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-07-10T09:11:54.638 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-07-10T09:11:54.639 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-07-10T09:11:54.848 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-07-10T09:11:55.050 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-07-10T09:11:55.073 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-07-10T09:11:55.254 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-07-10T09:11:55.276 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-07-10T09:11:55.278 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-07-10T09:11:55.465 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-07-10T09:11:55.498 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-07-10T09:11:55.679 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-07-10T09:11:55.789 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-07-10T09:11:55.922 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-07-10T09:11:55.965 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-07-10T09:11:56.156 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-07-10T09:11:56.210 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-10T09:11:56.390 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-07-10T09:11:56.444 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-07-10T09:11:56.601 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-07-10T09:11:56.665 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-07-10T09:11:56.817 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-07-10T09:11:56.819 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-07-10T09:11:56.881 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-07-10T09:11:57.019 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-07-10T09:11:57.104 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-07-10T09:11:57.230 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-07-10T09:11:57.315 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-07-10T09:11:57.431 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-07-10T09:11:57.535 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-07-10T09:11:57.641 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-07-10T09:11:57.747 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-07-10T09:11:57.852 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-07-10T09:11:57.854 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-07-10T09:11:57.968 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-07-10T09:11:58.063 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-07-10T09:11:58.066 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-07-10T09:11:58.189 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-07-10T09:11:58.274 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-07-10T09:11:58.412 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-07-10T09:11:58.487 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-07-10T09:11:58.625 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-07-10T09:11:58.689 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-07-10T09:11:58.838 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-07-10T09:11:58.902 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-07-10T09:11:59.113 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-07-10T09:11:59.251 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-07-10T09:11:59.326 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-07-10T09:11:59.473 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-07-10T09:11:59.526 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-07-10T09:11:59.684 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-07-10T09:11:59.748 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-07-10T09:11:59.750 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-07-10T09:11:59.906 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-07-10T09:11:59.948 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-07-10T09:12:00.117 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-07-10T09:12:00.119 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-07-10T09:12:00.159 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-07-10T09:12:00.338 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-07-10T09:12:00.371 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-07-10T09:12:00.571 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-07-10T09:12:00.583 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-07-10T09:12:00.784 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-07-10T09:12:00.795 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-07-10T09:12:00.995 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-07-10T09:12:00.997 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-07-10T09:12:01.251 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-07-10T09:12:01.253 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-07-10T09:12:01.263 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-07-10T09:12:01.265 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-10T09:12:01.513 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-07-10T09:12:01.515 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-07-10T09:12:01.517 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-07-10T09:12:01.739 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-07-10T09:12:01.750 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-07-10T09:12:01.973 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-07-10T09:12:01.995 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-07-10T09:12:02.179 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-07-10T09:12:02.244 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-07-10T09:12:02.391 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-07-10T09:12:02.465 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-07-10T09:12:02.593 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-07-10T09:12:02.689 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-07-10T09:12:02.806 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-07-10T09:12:02.902 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-07-10T09:12:03.018 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-07-10T09:12:03.123 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-07-10T09:12:03.219 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-07-10T09:12:03.346 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-07-10T09:12:03.348 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-07-10T09:12:03.442 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-07-10T09:12:03.559 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-07-10T09:12:03.643 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-07-10T09:12:03.783 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-07-10T09:12:03.858 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-07-10T09:12:04.008 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-07-10T09:12:04.072 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-07-10T09:12:04.230 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-07-10T09:12:04.283 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-07-10T09:12:04.452 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-07-10T09:12:04.496 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-07-10T09:12:04.666 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-07-10T09:12:04.698 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-07-10T09:12:04.883 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-07-10T09:12:04.885 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-10T09:12:04.916 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-07-10T09:12:05.105 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-07-10T09:12:05.117 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-07-10T09:12:05.328 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-07-10T09:12:05.330 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-07-10T09:12:05.533 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-07-10T09:12:05.544 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-07-10T09:12:05.742 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-07-10T09:12:05.765 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-07-10T09:12:05.954 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-07-10T09:12:05.976 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-07-10T09:12:06.155 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-07-10T09:12:06.198 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-07-10T09:12:06.366 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-07-10T09:12:06.420 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-07-10T09:12:06.566 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-07-10T09:12:06.640 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-07-10T09:12:06.777 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-07-10T09:12:06.851 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-07-10T09:12:06.988 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-07-10T09:12:07.072 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-07-10T09:12:07.187 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-07-10T09:12:07.292 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-07-10T09:12:07.397 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-07-10T09:12:07.512 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-07-10T09:12:07.596 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-07-10T09:12:07.732 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-07-10T09:12:07.805 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-07-10T09:12:07.951 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-07-10T09:12:08.015 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-07-10T09:12:08.161 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-07-10T09:12:08.214 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-07-10T09:12:08.381 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-07-10T09:12:08.433 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-07-10T09:12:08.601 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-07-10T09:12:08.633 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-07-10T09:12:08.821 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-07-10T09:12:08.842 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-07-10T09:12:08.844 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-07-10T09:12:09.033 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-07-10T09:12:09.035 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-07-10T09:12:09.045 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-07-10T09:12:09.257 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-07-10T09:12:09.258 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-07-10T09:12:09.469 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-07-10T09:12:09.480 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-07-10T09:12:09.679 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-07-10T09:12:09.701 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-07-10T09:12:09.890 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-07-10T09:12:09.924 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-07-10T09:12:10.092 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-07-10T09:12:10.146 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-07-10T09:12:10.304 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-07-10T09:12:10.357 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-07-10T09:12:10.515 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-07-10T09:12:10.579 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-07-10T09:12:10.727 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-07-10T09:12:10.801 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-07-10T09:12:10.928 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-07-10T09:12:11.023 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-07-10T09:12:11.139 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-07-10T09:12:11.235 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-07-10T09:12:11.350 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICYMwYvykHtnVhhJwcKaiRh9oMpJkqvEA7OAbHPYOv6W matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-07-10T09:12:11.455 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-07-10T09:12:11.560 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/Gyl/Xhn6OJ6mdvGv0a7LUMeEU6xDfecjod/OUf8yBFFLm/h0AMifHH8qQYoH9SkeUthXIggbi0fcx9nrgyqYgLOansWiEIPbBbVCpvBz1vT1UR0Bxg6B7bjcrMgKg3M1YhTiHt41PlxdW/nlhTlxyZ1+nM50Fu/WE1+4nVIUaTtyaKv8PnmLmnpDnNly9X9CPDyilx8xpNiZMTv7oaiilCIfnmrKaZEDuvpU0M6h5wa6wJCatxTGIt/RQo2c4/YL+Nmx0xkouoRwR1H6swhkZPbInFgy/IlmLuu+owFR5mCVdxmst04o18CyvlW//Wiepw/5uJA3Xivcu1hwwcesTO+xJgIQbM/hvJ/UmL/eJvnV9G9g1B7ttrVfITbYgXZ1gbtP4pJCUmq132o6q+qeyOsGl2k5f/rRl5Adch5lzepffSSfsYn2tS77b7uXf3ujGvwhlOwJRITRCDZ4nGFwijFwBg8ptFKTaXZsXjMQRtSP2kuHoRonmOm1XbD4ElM= root@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-07-10T09:12:11.676 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-07-10T09:12:11.771 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-07-10T09:12:11.888 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-07-10T09:12:11.973 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-07-10T09:12:12.100 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-07-10T09:12:12.185 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-07-10T09:12:12.285 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:12 +0000 (0:00:29.386) 0:01:32.085 ********* 2021-07-10T09:12:12.319 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-07-10T09:12:12.512 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:12.514 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.525 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.526 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg 2021-07-10T09:12:12.528 INFO:teuthology.task.ansible.out:8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-07-10T09:12:12.538 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:12.540 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.552 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.553 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.563 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.763 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:12.765 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-07-10T09:12:12.767 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": tr 2021-07-10T09:12:12.769 INFO:teuthology.task.ansible.out:ue, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-07-10T09:12:12.776 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.788 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:12.978 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-07-10T09:12:12.989 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:12.991 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-07-10T09:12:13.198 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:13.200 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi105.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4 2021-07-10T09:12:13.202 INFO:teuthology.task.ansible.out:a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.211 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.213 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.224 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.235 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.236 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.248 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.249 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.263 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.272 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.273 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.285 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.287 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.296 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.298 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.308 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.319 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.321 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.331 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.332 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.342 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.344 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.353 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.365 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.366 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.376 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.378 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.388 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.399 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.401 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.410 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.412 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKo 2021-07-10T09:12:13.414 INFO:teuthology.task.ansible.out:q8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.422 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.434 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.436 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.458 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.460 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.471 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.473 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.483 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.494 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.505 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.507 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.517 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.528 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.529 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.539 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.541 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.561 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.563 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.573 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.574 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.585 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.587 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.597 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.599 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.608 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.619 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.621 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.631 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:13.633 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "o 2021-07-10T09:12:13.634 INFO:teuthology.task.ansible.out:liveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.644 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.655 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.656 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.667 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.678 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.680 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.690 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.701 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.713 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.714 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.724 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.726 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.736 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.738 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.748 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.759 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.770 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.772 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.781 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.783 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.793 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.794 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.805 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.816 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.817 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.827 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.828 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.839 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.842 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-07-10T09:12:13.851 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:13.853 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.863 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.874 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.876 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.886 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.887 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.898 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.900 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.910 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.921 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.923 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.933 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.934 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.944 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.945 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.956 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.968 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.979 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.990 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:13.992 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.002 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.003 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.014 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.024 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.026 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.035 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.037 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.046 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.061 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_c 2021-07-10T09:12:14.062 INFO:teuthology.task.ansible.out:erts": true} 2021-07-10T09:12:14.070 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.073 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.083 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.085 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.094 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.105 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.107 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.116 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.118 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.127 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.138 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.179 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:14 +0000 (0:00:01.893) 0:01:33.979 ********* 2021-07-10T09:12:14.248 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-07-10T09:12:14.249 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:14.280 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:14 +0000 (0:00:00.100) 0:01:34.080 ********* 2021-07-10T09:12:14.290 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:14.292 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-07-10T09:12:14.492 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-07-10T09:12:14.556 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:14.558 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-07-10T09:12:14.715 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:14.717 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-07-10T09:12:14.747 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:14.749 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-07-10T09:12:14.933 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:14.934 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-07-10T09:12:14.944 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:14.946 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-07-10T09:12:15.134 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-07-10T09:12:15.156 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:15.158 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-07-10T09:12:15.377 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICYMwYvykHtnVhhJwcKaiRh9oMpJkqvEA7OAbHPYOv6W matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-07-10T09:12:15.462 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:15.464 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-07-10T09:12:15.598 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:15.600 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/Gyl/Xhn6OJ6mdvGv0a7LUMeEU6xDfecjod/OUf8yBFFLm/h0AMifHH8qQYoH9SkeUthXIggbi0fcx9nrgyqYgLOansWiEIPbBbVCpvBz1vT1UR0Bxg6B7bjcrMgKg3M1YhTiHt41PlxdW/nlhTlxyZ1+nM50Fu/WE1+4nVIUaTtyaKv8PnmLmnpDnNly9X9CPDyilx8xpNiZMTv7oaiilCIfnmrKaZEDuvpU0M6h5wa6wJCatxTGIt/RQo2c4/YL+Nmx0xkouoRwR1H6swhkZPbInFgy/IlmLuu+owFR5mCVdxmst04o18CyvlW//Wiepw/5uJA3Xivcu1hwwcesTO+xJgIQbM/hvJ/UmL/eJvnV9G9g1B7ttrVfITbYgXZ1gbtP4pJCUmq132o6q+qeyOsGl2k5f/rRl5Adch5lzepffSSfsYn2tS77b7uXf3ujGvwhlOwJRITRCDZ4nGFwijFwBg8ptFKTaXZsXjMQRtSP2kuHoRonmOm1XbD4ElM= root@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-07-10T09:12:15.651 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:15.653 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-07-10T09:12:15.841 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:15.843 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-07-10T09:12:16.039 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-07-10T09:12:16.071 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:16.073 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-07-10T09:12:16.230 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:16.232 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-07-10T09:12:16.294 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-10T09:12:16.298 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-07-10T09:12:16.525 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:16.526 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-07-10T09:12:16.546 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:16 +0000 (0:00:02.266) 0:01:36.346 ********* 2021-07-10T09:12:16.622 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-07-10T09:12:16.726 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6 2021-07-10T09:12:16.727 INFO:teuthology.task.ansible.out:/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52f 2021-07-10T09:12:16.729 INFO:teuthology.task.ansible.out:c0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:16.820 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi105.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-07-10T09:12:16.936 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:16.938 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-07-10T09:12:16.948 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:16.959 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.022 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:17.024 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-07-10T09:12:17.169 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:17.171 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-07-10T09:12:17.212 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:17.214 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-07-10T09:12:17.381 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi062.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-07-10T09:12:17.392 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.393 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.404 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi105.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:17.406 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.415 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.426 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.428 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.438 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.448 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.459 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.461 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.472 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.482 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.485 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.493 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.495 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.505 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.516 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.528 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.530 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.533 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.551 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.552 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.563 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.574 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.585 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.597 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi105.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-07-10T09:12:17.608 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:17.610 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.620 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.631 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.633 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.643 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.654 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.656 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.665 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.677 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.687 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.689 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.699 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.710 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.712 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.721 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.723 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.733 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.736 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.744 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.756 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.767 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.777 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.779 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.789 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.791 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi105.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.801 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.812 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.814 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.825 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.836 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.847 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.848 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.858 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.861 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.871 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.882 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.884 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.894 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.896 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.906 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.907 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.917 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.930 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.931 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.941 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.951 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.954 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.963 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.974 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.975 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.985 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.986 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi105.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.997 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:17.999 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.020 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.021 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.032 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.043 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.055 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.066 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.078 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.089 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.091 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.100 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.103 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.112 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.123 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.125 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.137 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.147 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.159 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.170 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.181 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.183 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi105.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-07-10T09:12:18.194 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-10T09:12:18.196 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.206 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.217 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.219 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.229 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.230 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.240 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.252 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.253 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.263 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.275 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.286 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.288 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.297 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.299 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.309 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.320 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.331 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.332 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.342 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.344 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.393 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:18 +0000 (0:00:01.846) 0:01:38.193 ********* 2021-07-10T09:12:18.400 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:18.402 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-07-10T09:12:18.476 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:18.498 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:18 +0000 (0:00:00.104) 0:01:38.298 ********* 2021-07-10T09:12:18.567 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:18.568 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-07-10T09:12:18.767 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-07-10T09:12:18.801 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-07-10T09:12:18.802 INFO:teuthology.task.ansible.out: 2021-07-10T09:12:18.962 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-07-10T09:12:18.984 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-07-10T09:12:19.153 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-07-10T09:12:19.185 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-07-10T09:12:19.344 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-07-10T09:12:19.387 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-07-10T09:12:19.534 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-07-10T09:12:19.587 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-07-10T09:12:19.726 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-07-10T09:12:19.790 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-07-10T09:12:19.928 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-07-10T09:12:19.992 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-07-10T09:12:20.118 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-07-10T09:12:20.203 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-07-10T09:12:20.318 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-07-10T09:12:20.403 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-07-10T09:12:20.509 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-07-10T09:12:20.605 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-07-10T09:12:20.701 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-07-10T09:12:20.806 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-07-10T09:12:20.890 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-07-10T09:12:21.006 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-07-10T09:12:21.090 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-07-10T09:12:21.216 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-07-10T09:12:21.290 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-07-10T09:12:21.416 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-07-10T09:12:21.481 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-07-10T09:12:21.616 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-07-10T09:12:21.669 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-07-10T09:12:21.828 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-07-10T09:12:21.872 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-07-10T09:12:22.042 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-07-10T09:12:22.118 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-07-10T09:12:22.265 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-07-10T09:12:22.321 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-07-10T09:12:22.489 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-07-10T09:12:22.553 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-07-10T09:12:22.701 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-07-10T09:12:22.765 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-07-10T09:12:22.902 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-07-10T09:12:22.977 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-07-10T09:12:23.116 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-07-10T09:12:23.170 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-07-10T09:12:23.317 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-07-10T09:12:23.381 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-07-10T09:12:23.528 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-07-10T09:12:23.571 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-07-10T09:12:23.729 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-07-10T09:12:23.772 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-07-10T09:12:23.931 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-07-10T09:12:23.963 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-07-10T09:12:24.135 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-07-10T09:12:24.169 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-07-10T09:12:24.224 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:24 +0000 (0:00:05.726) 0:01:44.024 ********* 2021-07-10T09:12:24.341 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-07-10T09:12:24.545 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-07-10T09:12:24.757 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-07-10T09:12:24.779 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "checksum": "946da8f37b852cb4beed25179833ebd51803d462", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "8cbbb9e68d3df7c0e0513064adc74bb8", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908344.3209143-4983-17855082184345/source", "state": "file", "uid": 0} 2021-07-10T09:12:24.807 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:24 +0000 (0:00:00.582) 0:01:44.607 ********* 2021-07-10T09:12:24.880 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi105.front.sepia.ceph.com] => { "msg": "Host smithi105.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-07-10T09:12:24.904 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:24 +0000 (0:00:00.096) 0:01:44.704 ********* 2021-07-10T09:12:24.947 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi062.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-07-10T09:12:24.979 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-07-10T09:12:24.980 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-07-10T09:12:25.013 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.109) 0:01:44.813 ********* 2021-07-10T09:12:25.082 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-07-10T09:12:25.084 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:25.106 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.092) 0:01:44.906 ********* 2021-07-10T09:12:25.165 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:25.165 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-07-10T09:12:25.355 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-07-10T09:12:25.441 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.109902", "end": "2021-07-10 09:12:25.422320", "rc": 0, "start": "2021-07-10 09:12:25.312418", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-07-10T09:12:25.462 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.356) 0:01:45.262 ********* 2021-07-10T09:12:25.531 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-07-10T09:12:25.532 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:25.555 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.092) 0:01:45.355 ********* 2021-07-10T09:12:25.562 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:25.564 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-07-10T09:12:25.627 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-07-10T09:12:25.628 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:25.649 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.094) 0:01:45.449 ********* 2021-07-10T09:12:25.719 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-07-10T09:12:25.721 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:25.741 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.091) 0:01:45.541 ********* 2021-07-10T09:12:25.757 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:25.759 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-07-10T09:12:25.815 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:25.836 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.095) 0:01:45.636 ********* 2021-07-10T09:12:25.916 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:25.939 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:25 +0000 (0:00:00.100) 0:01:45.737 ********* 2021-07-10T09:12:25.967 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:25.967 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-07-10T09:12:26.013 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-07-10T09:12:26.014 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:26.035 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.098) 0:01:45.835 ********* 2021-07-10T09:12:26.117 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi105.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi105.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:26.139 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.103) 0:01:45.939 ********* 2021-07-10T09:12:26.168 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:26.170 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-07-10T09:12:26.222 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:26.244 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.105) 0:01:46.044 ********* 2021-07-10T09:12:26.323 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-07-10T09:12:26.325 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:26.345 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.100) 0:01:46.145 ********* 2021-07-10T09:12:26.372 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:26.374 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-07-10T09:12:26.437 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-07-10T09:12:26.440 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:26.459 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.113) 0:01:46.259 ********* 2021-07-10T09:12:26.550 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi105.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:26.575 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.115) 0:01:46.375 ********* 2021-07-10T09:12:26.583 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:26.585 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-07-10T09:12:26.649 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:26.671 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.096) 0:01:46.471 ********* 2021-07-10T09:12:26.740 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-07-10T09:12:26.742 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:26.762 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.091) 0:01:46.562 ********* 2021-07-10T09:12:26.779 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:26.781 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-07-10T09:12:26.834 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:26.855 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.092) 0:01:46.655 ********* 2021-07-10T09:12:26.937 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:26.959 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:26 +0000 (0:00:00.103) 0:01:46.759 ********* 2021-07-10T09:12:26.986 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:26.988 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-07-10T09:12:27.029 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.052 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.092) 0:01:46.852 ********* 2021-07-10T09:12:27.130 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-07-10T09:12:27.132 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.153 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.101) 0:01:46.953 ********* 2021-07-10T09:12:27.192 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:27.194 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-07-10T09:12:27.223 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-07-10T09:12:27.225 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:27.245 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.092) 0:01:47.045 ********* 2021-07-10T09:12:27.324 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-10T09:12:27.325 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.347 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.102) 0:01:47.147 ********* 2021-07-10T09:12:27.388 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi062.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-07-10T09:12:27.420 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.442 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.095) 0:01:47.242 ********* 2021-07-10T09:12:27.523 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.547 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.103) 0:01:47.346 ********* 2021-07-10T09:12:27.594 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:27.596 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-07-10T09:12:27.627 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.648 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.102) 0:01:47.448 ********* 2021-07-10T09:12:27.729 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-07-10T09:12:27.731 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.754 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.105) 0:01:47.554 ********* 2021-07-10T09:12:27.794 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi062.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-07-10T09:12:27.828 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-07-10T09:12:27.830 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:27.852 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.097) 0:01:47.652 ********* 2021-07-10T09:12:27.922 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi105.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:27.947 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:27 +0000 (0:00:00.091) 0:01:47.744 ********* 2021-07-10T09:12:28.004 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi062.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-07-10T09:12:28.006 INFO:teuthology.task.ansible.out: 2021-07-10T09:12:28.016 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:28.038 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:28 +0000 (0:00:00.094) 0:01:47.838 ********* 2021-07-10T09:12:28.042 INFO:teuthology.task.ansible.out: 2021-07-10T09:12:28.118 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-07-10T09:12:28.120 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:28.141 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:28 +0000 (0:00:00.101) 0:01:47.940 ********* 2021-07-10T09:12:28.200 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:28.201 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-07-10T09:12:28.405 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} TASK [common : Increase the yum timeout.] ************************************** ok: [smithi105.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:12:28.426 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:28 +0000 (0:00:00.285) 0:01:48.226 ********* 2021-07-10T09:12:28.605 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-10T09:12:28.607 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-07-10T09:12:28.651 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:28 +0000 (0:00:00.224) 0:01:48.450 ********* 2021-07-10T09:12:29.240 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-07-10T09:12:29.241 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "checksum": "946da8f37b852cb4beed25179833ebd51803d462", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "8cbbb9e68d3df7c0e0513064adc74bb8", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908348.7480204-5436-243466392289293/source", "state": "file", "uid": 0} 2021-07-10T09:12:29.262 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:29 +0000 (0:00:00.611) 0:01:49.062 ********* 2021-07-10T09:12:29.311 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi105.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-07-10T09:12:29.333 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-07-10T09:12:29.335 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => { "msg": "Host smithi062.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-07-10T09:12:29.362 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:29 +0000 (0:00:00.099) 0:01:49.161 ********* 2021-07-10T09:12:29.456 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-07-10T09:12:29.458 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-07-10T09:12:29.479 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:29 +0000 (0:00:00.117) 0:01:49.279 ********* 2021-07-10T09:12:29.547 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-07-10T09:12:29.549 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:29.570 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:29 +0000 (0:00:00.091) 0:01:49.370 ********* 2021-07-10T09:12:29.776 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-07-10T09:12:29.777 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-07-10T09:12:29.802 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:29 +0000 (0:00:00.231) 0:01:49.602 ********* 2021-07-10T09:12:29.895 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-07-10T09:12:29.897 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.077133", "end": "2021-07-10 09:12:29.871681", "rc": 0, "start": "2021-07-10 09:12:29.794548", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-07-10T09:12:29.918 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:29 +0000 (0:00:00.116) 0:01:49.718 ********* 2021-07-10T09:12:30.014 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-07-10T09:12:30.017 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.037 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.118) 0:01:49.837 ********* 2021-07-10T09:12:30.116 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.138 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.101) 0:01:49.938 ********* 2021-07-10T09:12:30.218 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.240 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.102) 0:01:50.040 ********* 2021-07-10T09:12:30.320 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-07-10T09:12:30.322 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.346 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.105) 0:01:50.146 ********* 2021-07-10T09:12:30.365 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-07-10T09:12:30.367 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-07-10T09:12:30.420 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.443 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.097) 0:01:50.243 ********* 2021-07-10T09:12:30.525 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-07-10T09:12:30.527 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.547 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.103) 0:01:50.347 ********* 2021-07-10T09:12:30.639 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi062.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi062.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:30.665 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.118) 0:01:50.465 ********* 2021-07-10T09:12:30.666 INFO:teuthology.task.ansible.out: 2021-07-10T09:12:30.757 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.783 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.117) 0:01:50.583 ********* 2021-07-10T09:12:30.802 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-07-10T09:12:30.804 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-07-10T09:12:30.832 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.049) 0:01:50.632 ********* 2021-07-10T09:12:30.851 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-07-10T09:12:30.853 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.879 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.047) 0:01:50.679 ********* 2021-07-10T09:12:30.910 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.936 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.057) 0:01:50.736 ********* 2021-07-10T09:12:30.966 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-07-10T09:12:30.968 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:30.989 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:30 +0000 (0:00:00.052) 0:01:50.789 ********* 2021-07-10T09:12:31.018 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-10T09:12:31.020 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.043 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.053) 0:01:50.843 ********* 2021-07-10T09:12:31.084 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi062.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:31.086 INFO:teuthology.task.ansible.out: 2021-07-10T09:12:31.108 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.065) 0:01:50.908 ********* 2021-07-10T09:12:31.118 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-07-10T09:12:31.120 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.142 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.034) 0:01:50.942 ********* 2021-07-10T09:12:31.182 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-07-10T09:12:31.184 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.208 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.065) 0:01:51.008 ********* 2021-07-10T09:12:31.226 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-07-10T09:12:31.228 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.251 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.043) 0:01:51.051 ********* 2021-07-10T09:12:31.292 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi062.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:31.316 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.064) 0:01:51.116 ********* 2021-07-10T09:12:31.335 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-07-10T09:12:31.337 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.348 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.373 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.056) 0:01:51.173 ********* 2021-07-10T09:12:31.402 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-07-10T09:12:31.404 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.430 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.055) 0:01:51.228 ********* 2021-07-10T09:12:31.448 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-10T09:12:31.449 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.471 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.042) 0:01:51.271 ********* 2021-07-10T09:12:31.508 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-07-10T09:12:31.510 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.534 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.063) 0:01:51.334 ********* 2021-07-10T09:12:31.552 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-07-10T09:12:31.554 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.578 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.043) 0:01:51.378 ********* 2021-07-10T09:12:31.610 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.634 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.055) 0:01:51.434 ********* 2021-07-10T09:12:31.652 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.677 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.043) 0:01:51.477 ********* 2021-07-10T09:12:31.705 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.728 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.051) 0:01:51.528 ********* 2021-07-10T09:12:31.746 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.769 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.040) 0:01:51.569 ********* 2021-07-10T09:12:31.797 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-07-10T09:12:31.799 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:31.825 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.055) 0:01:51.625 ********* 2021-07-10T09:12:31.844 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.867 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.042) 0:01:51.667 ********* 2021-07-10T09:12:31.898 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-10T09:12:31.900 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:31.924 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:31 +0000 (0:00:00.056) 0:01:51.724 ********* 2021-07-10T09:12:31.996 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:32.021 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.096) 0:01:51.821 ********* 2021-07-10T09:12:32.101 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:32.124 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.102) 0:01:51.923 ********* 2021-07-10T09:12:32.193 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:32.216 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.092) 0:01:52.016 ********* 2021-07-10T09:12:32.298 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-07-10T09:12:32.299 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:32.321 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.104) 0:01:52.121 ********* 2021-07-10T09:12:32.400 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-07-10T09:12:32.402 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:32.423 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.102) 0:01:52.223 ********* 2021-07-10T09:12:32.502 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-07-10T09:12:32.504 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-10T09:12:32.526 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.102) 0:01:52.326 ********* 2021-07-10T09:12:32.533 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-07-10T09:12:32.535 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.460540", "end": "2021-07-10 09:12:32.500354", "rc": 0, "start": "2021-07-10 09:12:32.039814", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:12:32.558 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.032) 0:01:52.358 ********* 2021-07-10T09:12:32.597 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-07-10T09:12:32.598 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:32.620 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.061) 0:01:52.420 ********* 2021-07-10T09:12:32.701 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-07-10T09:12:32.702 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:32.723 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:32 +0000 (0:00:00.103) 0:01:52.523 ********* 2021-07-10T09:12:33.001 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-07-10T09:12:33.003 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:12:33.024 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:33 +0000 (0:00:00.300) 0:01:52.824 ********* 2021-07-10T09:12:33.608 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi062.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-07-10T09:12:34.061 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-07-10T09:12:34.084 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:34 +0000 (0:00:01.060) 0:01:53.884 ********* 2021-07-10T09:12:34.636 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-07-10T09:12:34.638 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-07-10T09:12:35.099 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-07-10T09:12:35.123 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:01.038) 0:01:54.923 ********* 2021-07-10T09:12:35.206 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-10T09:12:35.207 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.229 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.106) 0:01:55.029 ********* 2021-07-10T09:12:35.304 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-10T09:12:35.305 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.329 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.097) 0:01:55.127 ********* 2021-07-10T09:12:35.410 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-07-10T09:12:35.412 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.434 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.107) 0:01:55.234 ********* 2021-07-10T09:12:35.514 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-07-10T09:12:35.516 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.537 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.103) 0:01:55.337 ********* 2021-07-10T09:12:35.617 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi062.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.618 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.631 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.652 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.115) 0:01:55.452 ********* 2021-07-10T09:12:35.733 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-10T09:12:35.734 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.756 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.104) 0:01:55.556 ********* 2021-07-10T09:12:35.836 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-07-10T09:12:35.838 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.861 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.104) 0:01:55.661 ********* 2021-07-10T09:12:35.922 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-07-10T09:12:35.924 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:35.945 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:35 +0000 (0:00:00.084) 0:01:55.745 ********* 2021-07-10T09:12:35.947 INFO:teuthology.task.ansible.out: 2021-07-10T09:12:36.015 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:36.037 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.092) 0:01:55.837 ********* 2021-07-10T09:12:36.117 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-07-10T09:12:36.119 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:36.140 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.102) 0:01:55.940 ********* 2021-07-10T09:12:36.518 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-07-10T09:12:36.519 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.168789", "end": "2021-07-10 09:12:36.500389", "rc": 0, "start": "2021-07-10 09:12:36.331600", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:12:36.543 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.402) 0:01:56.342 ********* 2021-07-10T09:12:36.552 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-07-10T09:12:36.554 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:36.579 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.036) 0:01:56.379 ********* 2021-07-10T09:12:36.660 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-07-10T09:12:36.662 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:36.685 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.105) 0:01:56.485 ********* 2021-07-10T09:12:36.759 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:36.784 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.098) 0:01:56.584 ********* 2021-07-10T09:12:36.858 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:36.883 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.098) 0:01:56.683 ********* 2021-07-10T09:12:36.963 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:36.989 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:36 +0000 (0:00:00.105) 0:01:56.789 ********* 2021-07-10T09:12:37.510 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-07-10T09:12:37.511 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-07-10T09:12:37.534 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:37 +0000 (0:00:00.544) 0:01:57.333 ********* 2021-07-10T09:12:38.968 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-07-10T09:12:38.970 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:38.990 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:38 +0000 (0:00:01.457) 0:01:58.790 ********* 2021-07-10T09:12:39.563 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-07-10T09:12:39.565 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-07-10T09:12:40.008 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-07-10T09:12:40.327 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-07-10T09:12:40.329 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:40.359 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:40 +0000 (0:00:01.368) 0:02:00.159 ********* 2021-07-10T09:12:40.440 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-07-10T09:12:40.442 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:40.462 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:40 +0000 (0:00:00.103) 0:02:00.262 ********* 2021-07-10T09:12:40.470 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-07-10T09:12:40.472 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-07-10T09:12:40.499 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:40 +0000 (0:00:00.036) 0:02:00.299 ********* 2021-07-10T09:12:40.529 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:40.554 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:40 +0000 (0:00:00.054) 0:02:00.354 ********* 2021-07-10T09:12:40.625 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-07-10T09:12:40.627 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:40.649 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:40 +0000 (0:00:00.094) 0:02:00.449 ********* 2021-07-10T09:12:40.700 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:12:40.731 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:40 +0000 (0:00:00.082) 0:02:00.531 ********* 2021-07-10T09:12:40.738 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-07-10T09:12:40.740 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:40.767 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:40 +0000 (0:00:00.036) 0:02:00.567 ********* 2021-07-10T09:12:41.254 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-10T09:12:41.256 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-07-10T09:12:41.299 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-07-10T09:12:41.302 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-07-10T09:12:41.324 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:41 +0000 (0:00:00.556) 0:02:01.124 ********* 2021-07-10T09:12:41.679 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi105.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-07-10T09:12:42.163 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908361.7089493-7059-117564839495161/source", "state": "file", "uid": 0} 2021-07-10T09:12:42.191 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:42 +0000 (0:00:00.867) 0:02:01.991 ********* 2021-07-10T09:12:42.273 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-07-10T09:12:42.275 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:42.301 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:42 +0000 (0:00:00.109) 0:02:02.101 ********* 2021-07-10T09:12:42.403 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-07-10T09:12:42.405 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-07-10T09:12:42.426 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:42 +0000 (0:00:00.125) 0:02:02.226 ********* 2021-07-10T09:12:42.739 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-07-10T09:12:42.743 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-10T09:12:42.764 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:42 +0000 (0:00:00.338) 0:02:02.564 ********* 2021-07-10T09:12:42.814 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-07-10T09:12:42.816 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:42.837 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:42 +0000 (0:00:00.073) 0:02:02.637 ********* 2021-07-10T09:12:42.868 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi105.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:42.880 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:42.891 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:42.902 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:42.933 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:42 +0000 (0:00:00.095) 0:02:02.733 ********* 2021-07-10T09:12:43.410 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-07-10T09:12:43.412 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-07-10T09:12:43.871 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-07-10T09:12:44.333 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-07-10T09:12:44.356 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:44 +0000 (0:00:01.423) 0:02:04.156 ********* 2021-07-10T09:12:44.374 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-07-10T09:12:44.376 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:44.399 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:44 +0000 (0:00:00.042) 0:02:04.199 ********* 2021-07-10T09:12:44.480 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-07-10T09:12:44.482 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:44.503 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:44 +0000 (0:00:00.103) 0:02:04.303 ********* 2021-07-10T09:12:44.550 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-07-10T09:12:44.553 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:12:44.574 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:44 +0000 (0:00:00.071) 0:02:04.374 ********* 2021-07-10T09:12:45.019 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-07-10T09:12:45.020 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-07-10T09:12:45.042 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:45 +0000 (0:00:00.467) 0:02:04.842 ********* 2021-07-10T09:12:45.091 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-10T09:12:45.093 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-07-10T09:12:45.525 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-07-10T09:12:45.568 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-07-10T09:12:45.591 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:45 +0000 (0:00:00.549) 0:02:05.391 ********* 2021-07-10T09:12:45.699 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi105.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:45.708 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:45.730 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:45 +0000 (0:00:00.138) 0:02:05.530 ********* 2021-07-10T09:12:45.972 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-07-10T09:12:45.974 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:12:45.997 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:45 +0000 (0:00:00.266) 0:02:05.796 ********* 2021-07-10T09:12:46.049 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-10T09:12:46.051 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908365.5574899-7452-156502744270037/source", "state": "file", "uid": 0} 2021-07-10T09:12:46.082 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:46 +0000 (0:00:00.085) 0:02:05.882 ********* 2021-07-10T09:12:46.176 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-07-10T09:12:46.178 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:46.200 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:46 +0000 (0:00:00.117) 0:02:06.000 ********* 2021-07-10T09:12:46.293 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-07-10T09:12:46.295 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} TASK [common : Check firewalld status] ***************************************** fatal: [smithi105.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-07-10T09:12:46.319 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:46 +0000 (0:00:00.119) 0:02:06.119 ********* 2021-07-10T09:12:46.349 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:46 +0000 (0:00:00.029) 0:02:06.149 ********* 2021-07-10T09:12:46.438 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:46.473 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:46 +0000 (0:00:00.123) 0:02:06.273 ********* 2021-07-10T09:12:46.575 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi062.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-07-10T09:12:46.609 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:46 +0000 (0:00:00.135) 0:02:06.408 ********* 2021-07-10T09:12:46.710 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi062.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:46.732 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:46.743 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:46.754 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:46.777 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:46 +0000 (0:00:00.168) 0:02:06.577 ********* 2021-07-10T09:12:47.006 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-07-10T09:12:47.030 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:47 +0000 (0:00:00.252) 0:02:06.830 ********* 2021-07-10T09:12:47.862 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-07-10T09:12:47.864 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-10 09:10:20 UTC", "ActiveEnterTimestampMonotonic": "20312546", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target basic.target network-online.target sysinit.target var-run.mount systemd-tmpfiles-setup.service nss-lookup.target -.mount tmp.mount systemd-journald.socket system.slice local-fs.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-10 09:10:20 UTC", "AssertTimestampMonotonic": "20293535", "Before": "multi-user.target plymouth-quit.service getty@tty1.service xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-10 09:10:20 UTC", "ConditionT 2021-07-10T09:12:47.868 INFO:teuthology.task.ansible.out:imestampMonotonic": "20293535", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1237", "ExecMainStartTimestamp": "Sat 2021-07-10 09:10:20 UTC", "ExecMainStartTimestampMonotonic": "20312037", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-07-10 09:10:20 UTC] ; stop_time=[n/a] ; pid=1237 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-07-10 09:10:20 UTC", "InactiveExitTimestampMonotonic": "20312546", "In 2021-07-10T09:12:47.871 INFO:teuthology.task.ansible.out:vocationID": "a450fd7206e74f04bfb8d4b0c9abfe18", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1237", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGr 2021-07-10T09:12:47.874 INFO:teuthology.task.ansible.out:oups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-10 09:10:20 UTC", "StateChangeTimestampMonotonic": "20312546", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-10 09:10:20 UTC", "WatchdogTimestampMonotonic": "20312525", "WatchdogUSec": "0"}} 2021-07-10T09:12:47.888 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:47 +0000 (0:00:00.857) 0:02:07.687 ********* 2021-07-10T09:12:48.127 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-07-10T09:12:48.129 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001879", "end": "2021-07-10 09:12:48.102797", "rc": 0, "start": "2021-07-10 09:12:48.100918", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-07-10T09:12:48.153 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:48 +0000 (0:00:00.265) 0:02:07.953 ********* 2021-07-10T09:12:48.338 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-07-10T09:12:48.340 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:48.363 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:48 +0000 (0:00:00.208) 0:02:08.162 ********* 2021-07-10T09:12:48.435 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-07-10T09:12:48.437 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:48.458 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:48 +0000 (0:00:00.095) 0:02:08.258 ********* 2021-07-10T09:12:49.035 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-07-10T09:12:49.037 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-07-10T09:12:49.058 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:49 +0000 (0:00:00.600) 0:02:08.858 ********* 2021-07-10T09:12:49.571 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-07-10T09:12:49.572 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:49.598 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:49 +0000 (0:00:00.537) 0:02:09.396 ********* 2021-07-10T09:12:49.606 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-07-10T09:12:49.607 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-07-10T09:12:49.634 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:49 +0000 (0:00:00.037) 0:02:09.433 ********* 2021-07-10T09:12:49.718 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-07-10T09:12:49.720 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:49.730 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-07-10T09:12:49.754 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:49 +0000 (0:00:00.119) 0:02:09.553 ********* 2021-07-10T09:12:50.004 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi062.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:12:50.029 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:50 +0000 (0:00:00.276) 0:02:09.829 ********* 2021-07-10T09:12:50.313 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-07-10T09:12:50.315 INFO:teuthology.task.ansible.out:fatal: [smithi062.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-07-10T09:12:50.335 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:50 +0000 (0:00:00.305) 0:02:10.135 ********* 2021-07-10T09:12:50.415 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-07-10T09:12:50.417 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:12:50.439 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:50 +0000 (0:00:00.102) 0:02:10.238 ********* 2021-07-10T09:12:50.996 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-07-10T09:12:50.998 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-07-10T09:12:51.021 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:51 +0000 (0:00:00.583) 0:02:10.821 ********* 2021-07-10T09:12:51.399 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-07-10T09:12:51.401 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-10 09:10:19 UTC", "ActiveEnterTimestampMonotonic": "19713807", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target systemd-journald.socket time-sync.target system.slice tmp.mount nss-lookup.target sysinit.target -.mount local-fs.target systemd-tmpfiles-setup.service network-online.target var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-10 09:10:19 UTC", "AssertTimestampMonotonic": "19704476", "Before": "multi-user.target getty@tty1.service plymouth-quit.service xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-10 09:10:19 UTC", "ConditionT 2021-07-10T09:12:51.403 INFO:teuthology.task.ansible.out:imestampMonotonic": "19704475", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1238", "ExecMainStartTimestamp": "Sat 2021-07-10 09:10:19 UTC", "ExecMainStartTimestampMonotonic": "19713763", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-07-10 09:10:19 UTC] ; stop_time=[n/a] ; pid=1238 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-07-10 09:10:19 UTC", "InactiveExitTimestampMonotonic": "19713807", "In 2021-07-10T09:12:51.407 INFO:teuthology.task.ansible.out:vocationID": "5748be1a51a948dca9e6975a1a88ceea", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1238", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGr 2021-07-10T09:12:51.408 INFO:teuthology.task.ansible.out:oups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-10 09:10:19 UTC", "StateChangeTimestampMonotonic": "19713807", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-10 09:10:19 UTC", "WatchdogTimestampMonotonic": "19713805", "WatchdogUSec": "0"}} 2021-07-10T09:12:51.424 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:51 +0000 (0:00:00.402) 0:02:11.224 ********* 2021-07-10T09:12:51.663 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-07-10T09:12:51.665 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001880", "end": "2021-07-10 09:12:51.640219", "rc": 0, "start": "2021-07-10 09:12:51.638339", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-07-10T09:12:51.686 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:51 +0000 (0:00:00.261) 0:02:11.485 ********* 2021-07-10T09:12:53.179 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-07-10T09:12:53.181 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:12:53.201 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:12:53 +0000 (0:00:01.515) 0:02:13.001 ********* 2021-07-10T09:13:28.061 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-72.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-72.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-07-10T09:13:28.084 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:28 +0000 (0:00:34.882) 0:02:47.884 ********* 2021-07-10T09:13:28.876 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-07-10T09:13:28.898 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:28 +0000 (0:00:00.814) 0:02:48.698 ********* 2021-07-10T09:13:33.039 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-72.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-72.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-07-10T09:13:33.061 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:33 +0000 (0:00:04.162) 0:02:52.861 ********* 2021-07-10T09:13:33.585 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-07-10T09:13:33.587 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-07-10T09:13:33.610 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:33 +0000 (0:00:00.547) 0:02:53.408 ********* 2021-07-10T09:13:44.797 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.647333", "end": "2021-07-10 09:13:44.765497", "failed_when_result": false, "rc": 0, "start": "2021-07-10 09:13:29.118164", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-07-10T09:13:44.824 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:44 +0000 (0:00:11.212) 0:03:04.621 ********* 2021-07-10T09:13:45.399 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-07-10T09:13:45.402 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-07-10T09:13:45.421 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:45 +0000 (0:00:00.600) 0:03:05.221 ********* 2021-07-10T09:13:45.671 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-07-10T09:13:45.672 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013667", "end": "2021-07-10 09:13:45.652038", "rc": 0, "start": "2021-07-10 09:13:45.638371", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:13:45.706 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:45 +0000 (0:00:00.284) 0:03:05.506 ********* 2021-07-10T09:13:45.969 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-07-10T09:13:45.970 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013265", "end": "2021-07-10 09:13:45.942835", "rc": 0, "start": "2021-07-10 09:13:45.929570", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:13:46.001 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:45 +0000 (0:00:00.295) 0:03:05.801 ********* 2021-07-10T09:13:50.887 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-07-10T09:13:50.889 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:17.020969", "end": "2021-07-10 09:13:50.861338", "failed_when_result": false, "rc": 0, "start": "2021-07-10 09:13:33.840369", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-07-10T09:13:50.911 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:50 +0000 (0:00:04.909) 0:03:10.711 ********* 2021-07-10T09:13:51.435 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-07-10T09:13:51.437 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-07-10T09:13:51.469 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:51 +0000 (0:00:00.558) 0:03:11.269 ********* 2021-07-10T09:13:51.724 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018871", "end": "2021-07-10 09:13:51.704658", "rc": 0, "start": "2021-07-10 09:13:51.685787", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:13:51.746 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:51 +0000 (0:00:00.276) 0:03:11.546 ********* 2021-07-10T09:13:52.009 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.010693", "end": "2021-07-10 09:13:51.983977", "rc": 0, "start": "2021-07-10 09:13:51.973284", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:13:52.029 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:13:52 +0000 (0:00:00.283) 0:03:11.829 ********* 2021-07-10T09:14:02.012 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-07-10T09:14:02.014 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.769821", "end": "2021-07-10 09:14:01.988528", "rc": 0, "start": "2021-07-10 09:13:46.218707", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:14:02.036 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:02 +0000 (0:00:10.006) 0:03:21.836 ********* 2021-07-10T09:14:02.272 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-07-10T09:14:02.274 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-07-10T09:14:02.294 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:02 +0000 (0:00:00.258) 0:03:22.094 ********* 2021-07-10T09:14:02.373 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-07-10T09:14:02.375 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-07-10T09:14:02.396 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:02 +0000 (0:00:00.101) 0:03:22.196 ********* 2021-07-10T09:14:02.487 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-07-10T09:14:02.489 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-10T09:14:02.513 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:02 +0000 (0:00:00.116) 0:03:22.313 ********* 2021-07-10T09:14:02.593 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-07-10T09:14:02.594 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-07-10T09:14:02.616 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:02 +0000 (0:00:00.103) 0:03:22.416 ********* 2021-07-10T09:14:02.696 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-07-10T09:14:02.697 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-07-10T09:14:02.724 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:02 +0000 (0:00:00.108) 0:03:22.524 ********* 2021-07-10T09:14:02.918 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-07-10T09:14:02.920 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-10T09:14:02.943 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:02 +0000 (0:00:00.219) 0:03:22.743 ********* 2021-07-10T09:14:03.171 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-07-10T09:14:03.172 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-07-10T09:14:03.193 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:03 +0000 (0:00:00.249) 0:03:22.993 ********* 2021-07-10T09:14:03.262 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-07-10T09:14:03.263 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:03.284 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:03 +0000 (0:00:00.091) 0:03:23.084 ********* 2021-07-10T09:14:03.352 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-07-10T09:14:03.354 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:03.375 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:03 +0000 (0:00:00.090) 0:03:23.175 ********* 2021-07-10T09:14:03.642 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-07-10T09:14:03.644 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-07-10T09:14:03.666 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:03 +0000 (0:00:00.291) 0:03:23.466 ********* 2021-07-10T09:14:04.263 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-07-10T09:14:04.265 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-07-10T09:14:04.285 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:04 +0000 (0:00:00.619) 0:03:24.085 ********* 2021-07-10T09:14:04.364 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-07-10T09:14:04.366 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi105"}, "changed": false} 2021-07-10T09:14:04.386 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:04 +0000 (0:00:00.100) 0:03:24.186 ********* 2021-07-10T09:14:05.278 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi105.front.sepia.ceph.com", "ansible_hostname": "smithi105", "ansible_nodename": "smithi105"}, "changed": true, "name": "smithi105"} 2021-07-10T09:14:05.301 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:05 +0000 (0:00:00.914) 0:03:25.101 ********* 2021-07-10T09:14:05.849 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-07-10T09:14:05.851 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-07-10T09:14:05.872 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:05 +0000 (0:00:00.571) 0:03:25.672 ********* 2021-07-10T09:14:06.371 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-07-10T09:14:06.373 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-07-10T09:14:06.393 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:06 +0000 (0:00:00.521) 0:03:26.193 ********* 2021-07-10T09:14:06.951 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-07-10T09:14:06.953 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-07-10T09:14:06.973 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:06 +0000 (0:00:00.580) 0:03:26.773 ********* 2021-07-10T09:14:07.483 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-07-10T09:14:07.485 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-07-10T09:14:07.505 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:07 +0000 (0:00:00.531) 0:03:27.305 ********* 2021-07-10T09:14:07.745 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-10T09:14:07.766 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:07 +0000 (0:00:00.260) 0:03:27.566 ********* 2021-07-10T09:14:08.003 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-10T09:14:08.005 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-07-10T09:14:08.025 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:08 +0000 (0:00:00.259) 0:03:27.825 ********* 2021-07-10T09:14:08.294 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-10T09:14:08.296 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-07-10T09:14:08.316 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:08 +0000 (0:00:00.290) 0:03:28.116 ********* 2021-07-10T09:14:08.761 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-07-10T09:14:08.763 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.491121", "end": "2021-07-10 09:14:08.736924", "rc": 0, "start": "2021-07-10 09:13:52.245803", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:14:08.786 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:08 +0000 (0:00:00.470) 0:03:28.587 ********* 2021-07-10T09:14:08.827 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-07-10T09:14:08.851 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:08 +0000 (0:00:00.064) 0:03:28.651 ********* 2021-07-10T09:14:09.036 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-07-10T09:14:09.037 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-07-10T09:14:09.060 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.209) 0:03:28.860 ********* 2021-07-10T09:14:09.143 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-07-10T09:14:09.145 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-07-10T09:14:09.166 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.105) 0:03:28.966 ********* 2021-07-10T09:14:09.235 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-07-10T09:14:09.237 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-10T09:14:09.274 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.107) 0:03:29.073 ********* 2021-07-10T09:14:09.355 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-07-10T09:14:09.357 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-07-10T09:14:09.378 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.105) 0:03:29.178 ********* 2021-07-10T09:14:09.385 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-10T09:14:09.387 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-07-10T09:14:09.411 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.032) 0:03:29.211 ********* 2021-07-10T09:14:09.481 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-07-10T09:14:09.483 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-07-10T09:14:09.506 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.094) 0:03:29.306 ********* 2021-07-10T09:14:09.699 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-07-10T09:14:09.701 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-07-10T09:14:09.721 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.215) 0:03:29.521 ********* 2021-07-10T09:14:09.962 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-07-10T09:14:09.963 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-07-10T09:14:09.986 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:09 +0000 (0:00:00.264) 0:03:29.786 ********* 2021-07-10T09:14:10.057 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-07-10T09:14:10.058 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:10.079 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:10 +0000 (0:00:00.093) 0:03:29.879 ********* 2021-07-10T09:14:10.170 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-07-10T09:14:10.172 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:10.193 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:10 +0000 (0:00:00.113) 0:03:29.993 ********* 2021-07-10T09:14:10.493 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-07-10T09:14:10.495 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-07-10T09:14:10.516 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:10 +0000 (0:00:00.323) 0:03:30.316 ********* 2021-07-10T09:14:11.291 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-07-10T09:14:11.293 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-07-10T09:14:11.315 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:11 +0000 (0:00:00.799) 0:03:31.115 ********* 2021-07-10T09:14:11.409 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-07-10T09:14:11.413 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi062"}, "changed": false} 2021-07-10T09:14:11.443 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:11 +0000 (0:00:00.127) 0:03:31.242 ********* 2021-07-10T09:14:11.956 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-07-10T09:14:11.958 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi062.front.sepia.ceph.com", "ansible_hostname": "smithi062", "ansible_nodename": "smithi062"}, "changed": true, "name": "smithi062"} 2021-07-10T09:14:11.979 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:11 +0000 (0:00:00.536) 0:03:31.779 ********* 2021-07-10T09:14:12.041 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.369435", "end": "2021-07-10 09:14:12.016665", "rc": 0, "start": "2021-07-10 09:14:09.647230", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:14:12.065 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:12 +0000 (0:00:00.086) 0:03:31.865 ********* 2021-07-10T09:14:12.315 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ [WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-07-10T09:14:12.317 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*Stream*", "delta": "0:00:00.002830", "end": "2021-07-10 09:14:12.298704", "rc": 0, "start": "2021-07-10 09:14:12.295874", "stderr": "", "stderr_lines": [], "stdout": "removed '/etc/yum.repos.d/CentOS-AppStream-mirrorlist'\nremoved '/etc/yum.repos.d/CentOS-AppStream.repo'", "stdout_lines": ["removed '/etc/yum.repos.d/CentOS-AppStream-mirrorlist'", "removed '/etc/yum.repos.d/CentOS-AppStream.repo'"]} 2021-07-10T09:14:12.339 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:12 +0000 (0:00:00.273) 0:03:32.139 ********* 2021-07-10T09:14:12.419 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:12.421 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-07-10T09:14:12.443 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:12 +0000 (0:00:00.104) 0:03:32.243 ********* 2021-07-10T09:14:12.504 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-07-10T09:14:12.506 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-07-10T09:14:12.528 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:12 +0000 (0:00:00.084) 0:03:32.328 ********* 2021-07-10T09:14:13.023 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-07-10T09:14:13.024 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "md5sum": "a9798e07640f29320aff7f01b7ace08b", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908452.5531752-15792-56972484364982/source", "state": "file", "uid": 0} 2021-07-10T09:14:13.066 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-07-10T09:14:13.068 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-07-10T09:14:13.091 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:13 +0000 (0:00:00.562) 0:03:32.891 ********* 2021-07-10T09:14:13.465 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-07-10T09:14:13.466 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-07-10T09:14:13.694 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-07-10T09:14:13.696 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-07-10T09:14:13.718 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:13 +0000 (0:00:00.627) 0:03:33.519 ********* 2021-07-10T09:14:13.894 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-07-10T09:14:13.896 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-07-10T09:14:13.921 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:13 +0000 (0:00:00.202) 0:03:33.721 ********* 2021-07-10T09:14:14.246 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-07-10T09:14:14.248 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-07-10T09:14:14.270 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:14 +0000 (0:00:00.348) 0:03:34.070 ********* 2021-07-10T09:14:14.439 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-07-10T09:14:14.441 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-07-10T09:14:14.467 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:14 +0000 (0:00:00.196) 0:03:34.267 ********* 2021-07-10T09:14:14.515 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-07-10T09:14:14.517 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-10T09:14:14.539 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:14 +0000 (0:00:00.072) 0:03:34.339 ********* 2021-07-10T09:14:14.786 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-10T09:14:14.788 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-07-10T09:14:14.808 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:14 +0000 (0:00:00.269) 0:03:34.608 ********* 2021-07-10T09:14:15.033 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-10T09:14:15.035 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "md5sum": "70d010538a5a944c405442e0dbc3c454", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908454.572514-15966-240760503512786/source", "state": "file", "uid": 0} 2021-07-10T09:14:15.086 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-10T09:14:15.088 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-07-10T09:14:15.108 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:15 +0000 (0:00:00.299) 0:03:34.908 ********* 2021-07-10T09:14:15.478 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-10T09:14:15.480 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-07-10T09:14:15.637 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-10T09:14:15.638 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-07-10T09:14:15.659 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:15 +0000 (0:00:00.550) 0:03:35.459 ********* 2021-07-10T09:14:15.922 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-10T09:14:15.924 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-07-10T09:14:15.948 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:15 +0000 (0:00:00.289) 0:03:35.748 ********* 2021-07-10T09:14:16.018 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-07-10T09:14:16.041 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:16 +0000 (0:00:00.092) 0:03:35.841 ********* 2021-07-10T09:14:16.110 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-07-10T09:14:16.112 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:16.133 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:16 +0000 (0:00:00.092) 0:03:35.933 ********* 2021-07-10T09:14:16.193 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-10T09:14:16.195 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-07-10T09:14:16.217 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:16 +0000 (0:00:00.084) 0:03:36.017 ********* 2021-07-10T09:14:16.226 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-07-10T09:14:16.228 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:16.253 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:16 +0000 (0:00:00.035) 0:03:36.053 ********* 2021-07-10T09:14:16.701 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-07-10T09:14:16.703 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.212664", "end": "2021-07-10 09:14:16.678443", "rc": 0, "start": "2021-07-10 09:14:16.465779", "stderr": "", "stderr_lines": [], "stdout": "39 files removed", "stdout_lines": ["39 files removed"]} 2021-07-10T09:14:16.723 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:16 +0000 (0:00:00.470) 0:03:36.523 ********* 2021-07-10T09:14:16.803 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:16.805 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-07-10T09:14:16.825 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:16 +0000 (0:00:00.101) 0:03:36.625 ********* 2021-07-10T09:14:16.905 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-07-10T09:14:16.906 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:16.927 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:16 +0000 (0:00:00.101) 0:03:36.727 ********* 2021-07-10T09:14:17.007 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-07-10T09:14:17.008 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:17.030 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:17 +0000 (0:00:00.102) 0:03:36.829 ********* 2021-07-10T09:14:17.268 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-07-10T09:14:17.270 INFO:teuthology.task.ansible.out:fatal: [smithi105.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006598", "end": "2021-07-10 09:14:17.247191", "msg": "non-zero return code", "rc": 1, "start": "2021-07-10 09:14:17.240593", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-07-10T09:14:17.291 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:17 +0000 (0:00:00.261) 0:03:37.091 ********* 2021-07-10T09:14:17.371 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-07-10T09:14:17.373 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:17.394 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:17 +0000 (0:00:00.103) 0:03:37.194 ********* 2021-07-10T09:14:19.048 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-10T09:14:19.050 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.594621", "end": "2021-07-10 09:14:19.023248", "rc": 0, "start": "2021-07-10 09:14:16.428627", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:14:19.073 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:19 +0000 (0:00:01.678) 0:03:38.873 ********* 2021-07-10T09:14:19.360 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-07-10T09:14:19.362 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*Stream*", "delta": "0:00:00.003192", "end": "2021-07-10 09:14:19.333102", "rc": 0, "start": "2021-07-10 09:14:19.329910", "stderr": "", "stderr_lines": [], "stdout": "removed '/etc/yum.repos.d/CentOS-AppStream-mirrorlist'\nremoved '/etc/yum.repos.d/CentOS-AppStream.repo'", "stdout_lines": ["removed '/etc/yum.repos.d/CentOS-AppStream-mirrorlist'", "removed '/etc/yum.repos.d/CentOS-AppStream.repo'"]} 2021-07-10T09:14:19.387 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:19 +0000 (0:00:00.313) 0:03:39.186 ********* 2021-07-10T09:14:19.467 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:19.469 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-07-10T09:14:19.502 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:19 +0000 (0:00:00.115) 0:03:39.302 ********* 2021-07-10T09:14:20.116 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-07-10T09:14:20.118 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "md5sum": "a9798e07640f29320aff7f01b7ace08b", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908459.6185591-16471-146392216869384/source", "state": "file", "uid": 0} 2021-07-10T09:14:20.573 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-07-10T09:14:21.032 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-07-10T09:14:21.061 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:21 +0000 (0:00:01.558) 0:03:40.861 ********* 2021-07-10T09:14:21.609 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-07-10T09:14:21.611 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-07-10T09:14:21.636 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:21 +0000 (0:00:00.575) 0:03:41.437 ********* 2021-07-10T09:14:22.238 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-10T09:14:22.241 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "md5sum": "70d010538a5a944c405442e0dbc3c454", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908461.7418368-16743-254047054996502/source", "state": "file", "uid": 0} 2021-07-10T09:14:22.703 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-07-10T09:14:23.175 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-07-10T09:14:23.201 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:23 +0000 (0:00:01.563) 0:03:43.000 ********* 2021-07-10T09:14:23.280 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-07-10T09:14:23.303 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:23 +0000 (0:00:00.103) 0:03:43.103 ********* 2021-07-10T09:14:23.375 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-07-10T09:14:23.377 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:23.399 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:23 +0000 (0:00:00.095) 0:03:43.199 ********* 2021-07-10T09:14:23.505 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi062.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:23.507 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:23.531 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:23 +0000 (0:00:00.131) 0:03:43.331 ********* 2021-07-10T09:14:24.000 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.226170", "end": "2021-07-10 09:14:23.981334", "rc": 0, "start": "2021-07-10 09:14:23.755164", "stderr": "", "stderr_lines": [], "stdout": "39 files removed", "stdout_lines": ["39 files removed"]} 2021-07-10T09:14:24.022 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:24 +0000 (0:00:00.491) 0:03:43.822 ********* 2021-07-10T09:14:24.102 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:24.104 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-07-10T09:14:24.129 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:24 +0000 (0:00:00.107) 0:03:43.929 ********* 2021-07-10T09:14:24.207 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-07-10T09:14:24.209 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:24.233 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:24 +0000 (0:00:00.103) 0:03:44.033 ********* 2021-07-10T09:14:24.314 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:24.336 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:24 +0000 (0:00:00.103) 0:03:44.136 ********* 2021-07-10T09:14:24.582 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi062.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006938", "end": "2021-07-10 09:14:24.562978", "msg": "non-zero return code", "rc": 1, "start": "2021-07-10 09:14:24.556040", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-07-10T09:14:24.604 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:24 +0000 (0:00:00.268) 0:03:44.404 ********* 2021-07-10T09:14:24.673 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-07-10T09:14:24.675 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:24.697 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:24 +0000 (0:00:00.093) 0:03:44.497 ********* 2021-07-10T09:14:27.157 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-10T09:14:27.158 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-07-10T09:14:27.180 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:27 +0000 (0:00:02.482) 0:03:46.980 ********* 2021-07-10T09:14:29.706 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-10T09:14:29.708 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-07-10T09:14:29.729 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:29 +0000 (0:00:02.549) 0:03:49.529 ********* 2021-07-10T09:14:35.101 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-10T09:14:35.103 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-07-10T09:14:35.125 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:35 +0000 (0:00:05.396) 0:03:54.925 ********* 2021-07-10T09:14:37.720 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-10T09:14:37.722 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-07-10T09:14:37.743 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:37 +0000 (0:00:02.617) 0:03:57.543 ********* 2021-07-10T09:14:37.992 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-10T09:14:37.994 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: yum-utils-4.0.21-1.el8.noarch", "Installed: qemu-img-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-1.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: dnf-plugins-core-4.0.21-1.el8.noarch", "Installed: qemu-kvm-block-rbd-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Installed: qemu-kvm-common-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Installed: qemu-kvm-core-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-07-10T09:14:38.016 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:38 +0000 (0:00:00.273) 0:03:57.816 ********* 2021-07-10T09:14:38.086 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-07-10T09:14:38.087 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:38.112 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:38 +0000 (0:00:00.096) 0:03:57.913 ********* 2021-07-10T09:14:39.632 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-10T09:14:39.633 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:14:39.654 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:39 +0000 (0:00:01.541) 0:03:59.454 ********* 2021-07-10T09:14:42.403 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-10T09:14:42.405 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: libgcrypt-1.8.5-6.el8.x86_64", "Removed: libgcrypt-1.8.5-4.el8.x86_64"]} 2021-07-10T09:14:42.425 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:42 +0000 (0:00:02.771) 0:04:02.225 ********* 2021-07-10T09:14:42.779 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-07-10T09:14:42.780 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service system.slice dbus.socket polkit.service basic.target sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nftables.service ebtables.service ipset.service iptables.service ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", 2021-07-10T09:14:42.782 INFO:teuthology.task.ansible.out:"Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": " 2021-07-10T09:14:42.784 INFO:teuthology.task.ansible.out:0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null 2021-07-10T09:14:42.785 INFO:teuthology.task.ansible.out:", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-10T09:14:42.804 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:42 +0000 (0:00:00.378) 0:04:02.604 ********* 2021-07-10T09:14:43.127 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-07-10T09:14:43.129 INFO:teuthology.task.ansible.out:fatal: [smithi105.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-07-10T09:14:43.157 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:43 +0000 (0:00:00.353) 0:04:02.957 ********* 2021-07-10T09:14:43.932 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-07-10T09:14:43.933 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-07-10T09:14:43.955 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:43 +0000 (0:00:00.797) 0:04:03.755 ********* 2021-07-10T09:14:44.024 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.046 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.091) 0:04:03.846 ********* 2021-07-10T09:14:44.116 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-07-10T09:14:44.118 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.147 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.100) 0:04:03.947 ********* 2021-07-10T09:14:44.226 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-07-10T09:14:44.228 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.249 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.102) 0:04:04.049 ********* 2021-07-10T09:14:44.328 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.350 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.101) 0:04:04.150 ********* 2021-07-10T09:14:44.431 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-07-10T09:14:44.433 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.455 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.105) 0:04:04.255 ********* 2021-07-10T09:14:44.528 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.553 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.096) 0:04:04.352 ********* 2021-07-10T09:14:44.621 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-07-10T09:14:44.643 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.091) 0:04:04.443 ********* 2021-07-10T09:14:44.713 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-07-10T09:14:44.715 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.734 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.091) 0:04:04.534 ********* 2021-07-10T09:14:44.803 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-07-10T09:14:44.805 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.827 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.092) 0:04:04.627 ********* 2021-07-10T09:14:44.907 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi105.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:44.932 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:44 +0000 (0:00:00.104) 0:04:04.732 ********* 2021-07-10T09:14:45.005 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-07-10T09:14:45.006 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.028 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.095) 0:04:04.827 ********* 2021-07-10T09:14:45.098 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-07-10T09:14:45.121 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.093) 0:04:04.921 ********* 2021-07-10T09:14:45.191 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.213 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.092) 0:04:05.013 ********* 2021-07-10T09:14:45.284 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-07-10T09:14:45.286 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.306 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.092) 0:04:05.106 ********* 2021-07-10T09:14:45.375 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-07-10T09:14:45.377 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.399 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.092) 0:04:05.199 ********* 2021-07-10T09:14:45.469 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-10T09:14:45.471 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.491 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.092) 0:04:05.291 ********* 2021-07-10T09:14:45.562 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-07-10T09:14:45.563 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.584 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.092) 0:04:05.384 ********* 2021-07-10T09:14:45.656 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.678 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.093) 0:04:05.478 ********* 2021-07-10T09:14:45.680 INFO:teuthology.task.ansible.out: 2021-07-10T09:14:45.770 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-07-10T09:14:45.772 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.792 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.114) 0:04:05.592 ********* 2021-07-10T09:14:45.863 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-07-10T09:14:45.865 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.885 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.092) 0:04:05.685 ********* 2021-07-10T09:14:45.957 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-07-10T09:14:45.959 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:45.986 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:45 +0000 (0:00:00.101) 0:04:05.786 ********* 2021-07-10T09:14:46.059 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-07-10T09:14:46.061 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.082 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.095) 0:04:05.882 ********* 2021-07-10T09:14:46.152 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-07-10T09:14:46.155 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.175 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.093) 0:04:05.975 ********* 2021-07-10T09:14:46.245 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-07-10T09:14:46.247 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.268 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.093) 0:04:06.069 ********* 2021-07-10T09:14:46.339 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install packages] ********************************************* 2021-07-10T09:14:46.341 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: yum-utils-4.0.21-1.el8.noarch", "Installed: qemu-img-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-1.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: dnf-plugins-core-4.0.21-1.el8.noarch", "Installed: qemu-kvm-block-rbd-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Installed: qemu-kvm-common-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Installed: qemu-kvm-core-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-07-10T09:14:46.361 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.092) 0:04:06.161 ********* 2021-07-10T09:14:46.381 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.020) 0:04:06.181 ********* 2021-07-10T09:14:46.430 INFO:teuthology.task.ansible.out: 2021-07-10T09:14:46.432 INFO:teuthology.task.ansible.out:TASK [testnode : Install epel packages] **************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.454 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.073) 0:04:06.255 ********* 2021-07-10T09:14:46.461 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-07-10T09:14:46.463 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.485 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.030) 0:04:06.285 ********* 2021-07-10T09:14:46.554 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.577 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.091) 0:04:06.377 ********* 2021-07-10T09:14:46.647 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.669 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.091) 0:04:06.469 ********* 2021-07-10T09:14:46.739 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-10T09:14:46.741 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.762 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.093) 0:04:06.562 ********* 2021-07-10T09:14:46.832 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-10T09:14:46.834 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.855 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.092) 0:04:06.655 ********* 2021-07-10T09:14:46.928 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-10T09:14:46.930 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:46.950 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:46 +0000 (0:00:00.094) 0:04:06.750 ********* 2021-07-10T09:14:47.019 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-10T09:14:47.021 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.041 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.091) 0:04:06.841 ********* 2021-07-10T09:14:47.111 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-10T09:14:47.113 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.136 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.094) 0:04:06.936 ********* 2021-07-10T09:14:47.204 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-10T09:14:47.206 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.227 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.090) 0:04:07.026 ********* 2021-07-10T09:14:47.295 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-10T09:14:47.298 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.318 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.091) 0:04:07.118 ********* 2021-07-10T09:14:47.387 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:47.389 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.409 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.091) 0:04:07.210 ********* 2021-07-10T09:14:47.480 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-10T09:14:47.481 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.504 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.094) 0:04:07.304 ********* 2021-07-10T09:14:47.574 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-10T09:14:47.576 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.598 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.094) 0:04:07.398 ********* 2021-07-10T09:14:47.670 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-10T09:14:47.672 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.692 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.094) 0:04:07.492 ********* 2021-07-10T09:14:47.763 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:47.792 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:47 +0000 (0:00:00.099) 0:04:07.592 ********* 2021-07-10T09:14:48.024 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-10T09:14:48.026 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:14:48.048 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:48 +0000 (0:00:00.255) 0:04:07.848 ********* 2021-07-10T09:14:48.054 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-10T09:14:48.056 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-07-10T09:14:48.081 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:48 +0000 (0:00:00.032) 0:04:07.881 ********* 2021-07-10T09:14:48.349 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-10T09:14:48.351 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:14:48.373 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:48 +0000 (0:00:00.291) 0:04:08.173 ********* 2021-07-10T09:14:48.613 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-10T09:14:48.615 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-07-10T09:14:48.636 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:48 +0000 (0:00:00.263) 0:04:08.436 ********* 2021-07-10T09:14:48.716 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:48.741 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:48 +0000 (0:00:00.104) 0:04:08.541 ********* 2021-07-10T09:14:48.812 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-10T09:14:48.813 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:48.834 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:48 +0000 (0:00:00.093) 0:04:08.634 ********* 2021-07-10T09:14:48.908 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-10T09:14:48.910 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:48.931 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:48 +0000 (0:00:00.096) 0:04:08.731 ********* 2021-07-10T09:14:49.002 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-10T09:14:49.004 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.025 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.093) 0:04:08.825 ********* 2021-07-10T09:14:49.095 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-10T09:14:49.097 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.118 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.093) 0:04:08.918 ********* 2021-07-10T09:14:49.190 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-10T09:14:49.192 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.216 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.097) 0:04:09.015 ********* 2021-07-10T09:14:49.296 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-10T09:14:49.298 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.318 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.103) 0:04:09.119 ********* 2021-07-10T09:14:49.391 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.415 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.094) 0:04:09.214 ********* 2021-07-10T09:14:49.483 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-10T09:14:49.486 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.506 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.092) 0:04:09.306 ********* 2021-07-10T09:14:49.599 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-07-10T09:14:49.601 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.623 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.117) 0:04:09.423 ********* 2021-07-10T09:14:49.695 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-07-10T09:14:49.696 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.718 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.095) 0:04:09.518 ********* 2021-07-10T09:14:49.789 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.811 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.093) 0:04:09.611 ********* 2021-07-10T09:14:49.881 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-10T09:14:49.883 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:49.907 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.096) 0:04:09.707 ********* 2021-07-10T09:14:49.978 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-07-10T09:14:49.980 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.000 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:49 +0000 (0:00:00.092) 0:04:09.800 ********* 2021-07-10T09:14:50.072 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-07-10T09:14:50.074 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.094 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.094) 0:04:09.894 ********* 2021-07-10T09:14:50.164 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.186 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.092) 0:04:09.986 ********* 2021-07-10T09:14:50.251 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-07-10T09:14:50.253 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.275 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.088) 0:04:10.075 ********* 2021-07-10T09:14:50.346 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-07-10T09:14:50.348 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.374 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.098) 0:04:10.174 ********* 2021-07-10T09:14:50.456 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.479 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.105) 0:04:10.279 ********* 2021-07-10T09:14:50.562 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.587 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.107) 0:04:10.387 ********* 2021-07-10T09:14:50.659 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.682 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.094) 0:04:10.482 ********* 2021-07-10T09:14:50.755 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.779 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.096) 0:04:10.578 ********* 2021-07-10T09:14:50.858 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:50.860 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-07-10T09:14:50.881 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:50 +0000 (0:00:00.103) 0:04:10.681 ********* 2021-07-10T09:14:50.973 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-07-10T09:14:50.974 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.986 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-07-10T09:14:50.996 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:50.999 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: libgcrypt-1.8.5-6.el8.x86_64", "Removed: libgcrypt-1.8.5-4.el8.x86_64"]} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi105.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:51.022 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.140) 0:04:10.822 ********* 2021-07-10T09:14:51.030 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:51.032 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:51.061 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.038) 0:04:10.861 ********* 2021-07-10T09:14:51.313 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003401", "end": "2021-07-10 09:14:51.291435", "rc": 0, "start": "2021-07-10 09:14:51.288034", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/virtualenv/bin/teuthology-lock"]} 2021-07-10T09:14:51.338 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.277) 0:04:11.138 ********* 2021-07-10T09:14:51.389 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-07-10T09:14:51.390 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target dbus.socket sysinit.target polkit.service system.slice dbus.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service shutdown.target iptables.service nftables.service ip6tables.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", 2021-07-10T09:14:51.391 INFO:teuthology.task.ansible.out:"Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": " 2021-07-10T09:14:51.393 INFO:teuthology.task.ansible.out:0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null 2021-07-10T09:14:51.395 INFO:teuthology.task.ansible.out:", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-10T09:14:51.414 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.076) 0:04:11.214 ********* 2021-07-10T09:14:51.420 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-07-10T09:14:51.422 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:51.447 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.033) 0:04:11.247 ********* 2021-07-10T09:14:51.520 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-07-10T09:14:51.521 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:51.543 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.095) 0:04:11.343 ********* 2021-07-10T09:14:51.614 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-07-10T09:14:51.616 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:51.637 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.093) 0:04:11.436 ********* 2021-07-10T09:14:51.707 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-07-10T09:14:51.708 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:51.729 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.092) 0:04:11.529 ********* 2021-07-10T09:14:51.757 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-07-10T09:14:51.759 INFO:teuthology.task.ansible.out:fatal: [smithi062.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-07-10T09:14:51.783 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:51 +0000 (0:00:00.053) 0:04:11.583 ********* 2021-07-10T09:14:52.115 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-07-10T09:14:52.137 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.354) 0:04:11.937 ********* 2021-07-10T09:14:52.241 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:52.262 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.125) 0:04:12.063 ********* 2021-07-10T09:14:52.344 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-07-10T09:14:52.346 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:52.370 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.107) 0:04:12.170 ********* 2021-07-10T09:14:52.462 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:52.485 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.114) 0:04:12.285 ********* 2021-07-10T09:14:52.577 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-07-10T09:14:52.579 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:52.599 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.114) 0:04:12.399 ********* 2021-07-10T09:14:52.670 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:52.695 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.096) 0:04:12.495 ********* 2021-07-10T09:14:52.764 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-07-10T09:14:52.766 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:52.787 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.091) 0:04:12.587 ********* 2021-07-10T09:14:52.856 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-07-10T09:14:52.881 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.093) 0:04:12.681 ********* 2021-07-10T09:14:52.950 INFO:teuthology.task.ansible.out: 2021-07-10T09:14:52.952 INFO:teuthology.task.ansible.out:TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:52.973 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:52 +0000 (0:00:00.092) 0:04:12.773 ********* 2021-07-10T09:14:53.036 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-07-10T09:14:53.036 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.059 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.086) 0:04:12.859 ********* 2021-07-10T09:14:53.129 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-07-10T09:14:53.131 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.140 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.164 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.104) 0:04:12.964 ********* 2021-07-10T09:14:53.235 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-07-10T09:14:53.237 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.258 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.094) 0:04:13.058 ********* 2021-07-10T09:14:53.328 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-07-10T09:14:53.350 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.092) 0:04:13.150 ********* 2021-07-10T09:14:53.421 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.443 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.092) 0:04:13.243 ********* 2021-07-10T09:14:53.451 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-07-10T09:14:53.451 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:14:53.477 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.033) 0:04:13.277 ********* 2021-07-10T09:14:53.520 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.543 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.065) 0:04:13.343 ********* 2021-07-10T09:14:53.572 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi105.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930170159104, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227092324, 'block_used': 13002784, 'inode_total': 61054976, 'inode_available': 60984981, 'inode_used': 69995, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227092324, "block_size": 4096, "block_total": 240095108, "block_used": 13002784, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984981, "inode_total": 61054976, "inode_used": 69995, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930170159104, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-07-10T09:14:53.597 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.053) 0:04:13.397 ********* 2021-07-10T09:14:53.615 INFO:teuthology.task.ansible.out: 2021-07-10T09:14:53.617 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.641 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.044) 0:04:13.441 ********* 2021-07-10T09:14:53.712 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.735 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.094) 0:04:13.535 ********* 2021-07-10T09:14:53.818 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-07-10T09:14:53.820 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.844 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.108) 0:04:13.644 ********* 2021-07-10T09:14:53.852 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-07-10T09:14:53.853 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005246", "end": "2021-07-10 09:14:53.814474", "rc": 0, "start": "2021-07-10 09:14:53.809228", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:14:53.878 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.033) 0:04:13.678 ********* 2021-07-10T09:14:53.929 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:53.955 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.077) 0:04:13.755 ********* 2021-07-10T09:14:53.977 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-07-10T09:14:53.980 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930170159104, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227092324, 'block_used': 13002784, 'inode_total': 61054976, 'inode_available': 60984981, 'inode_used': 69995, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227092324, "block_size": 4096, "block_total": 240095108, "block_used": 13002784, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984981, "inode_total": 61054976, "inode_used": 69995, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930170159104, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.006 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:53 +0000 (0:00:00.049) 0:04:13.805 ********* 2021-07-10T09:14:54.037 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.061 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.055) 0:04:13.861 ********* 2021-07-10T09:14:54.079 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-07-10T09:14:54.102 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.040) 0:04:13.902 ********* 2021-07-10T09:14:54.131 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.161 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.059) 0:04:13.961 ********* 2021-07-10T09:14:54.232 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-07-10T09:14:54.234 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.256 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.095) 0:04:14.056 ********* 2021-07-10T09:14:54.328 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-07-10T09:14:54.330 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.351 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.094) 0:04:14.151 ********* 2021-07-10T09:14:54.423 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.447 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.095) 0:04:14.247 ********* 2021-07-10T09:14:54.516 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-07-10T09:14:54.518 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.540 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.092) 0:04:14.339 ********* 2021-07-10T09:14:54.609 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-07-10T09:14:54.611 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.631 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.092) 0:04:14.432 ********* 2021-07-10T09:14:54.702 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.724 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.092) 0:04:14.524 ********* 2021-07-10T09:14:54.807 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-07-10T09:14:54.809 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.830 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.105) 0:04:14.630 ********* 2021-07-10T09:14:54.900 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:54.925 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:54 +0000 (0:00:00.092) 0:04:14.722 ********* 2021-07-10T09:14:54.996 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.021 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.098) 0:04:14.821 ********* 2021-07-10T09:14:55.089 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-10T09:14:55.091 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.112 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.090) 0:04:14.911 ********* 2021-07-10T09:14:55.180 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-10T09:14:55.182 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.202 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.090) 0:04:15.002 ********* 2021-07-10T09:14:55.272 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-10T09:14:55.273 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.294 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.092) 0:04:15.094 ********* 2021-07-10T09:14:55.365 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-10T09:14:55.366 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.387 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.092) 0:04:15.187 ********* 2021-07-10T09:14:55.458 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-10T09:14:55.460 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.482 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.094) 0:04:15.281 ********* 2021-07-10T09:14:55.556 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.578 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.097) 0:04:15.378 ********* 2021-07-10T09:14:55.648 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:55.650 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.671 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.092) 0:04:15.471 ********* 2021-07-10T09:14:55.700 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:14:55.701 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXs0vji8GIQchT4HdQszVzrDJurJ5u4bFB'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.325707", "end": "2021-07-10 09:14:55.662278", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXs0vji8GIQchT4HdQszVzrDJurJ5u4bFB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:14:54.336571", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:14:55.744 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.766 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.094) 0:04:15.566 ********* 2021-07-10T09:14:55.836 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-10T09:14:55.838 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.858 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.092) 0:04:15.658 ********* 2021-07-10T09:14:55.861 INFO:teuthology.task.ansible.out: 2021-07-10T09:14:55.931 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:55.955 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:55 +0000 (0:00:00.095) 0:04:15.754 ********* 2021-07-10T09:14:56.035 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-10T09:14:56.037 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:56.057 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:56 +0000 (0:00:00.103) 0:04:15.857 ********* 2021-07-10T09:14:56.317 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-10T09:14:56.319 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-07-10T09:14:56.343 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:56 +0000 (0:00:00.285) 0:04:16.143 ********* 2021-07-10T09:14:56.602 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-10T09:14:56.604 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:14:56.625 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:56 +0000 (0:00:00.282) 0:04:16.425 ********* 2021-07-10T09:14:56.876 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-07-10T09:14:56.899 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:56 +0000 (0:00:00.273) 0:04:16.699 ********* 2021-07-10T09:14:56.927 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:14:56.929 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-x41AQA-WTmf-Tdu4-h6DN-aVds-xClH-p6Bqd7', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400TR400BGN', 'nvme-nvme.8086-50484654363230343030545234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.046545", "end": "2021-07-10 09:14:56.901737", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-x41AQA-WTmf-Tdu4-h6DN-aVds-xClH-p6Bqd7", "nvme-INTEL_SSDPEDMD400G4_PHFT620400TR400BGN", "nvme-nvme.8086-50484654363230343030545234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:14:55.855192", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-07-10T09:14:56.931 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:14:56.971 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:56.994 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:56 +0000 (0:00:00.095) 0:04:16.794 ********* 2021-07-10T09:14:57.064 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-10T09:14:57.066 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.090 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.095) 0:04:16.890 ********* 2021-07-10T09:14:57.161 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.184 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.094) 0:04:16.984 ********* 2021-07-10T09:14:57.254 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-10T09:14:57.256 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.277 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.092) 0:04:17.077 ********* 2021-07-10T09:14:57.368 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-10T09:14:57.369 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.397 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.119) 0:04:17.196 ********* 2021-07-10T09:14:57.467 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-10T09:14:57.469 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.489 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.092) 0:04:17.289 ********* 2021-07-10T09:14:57.559 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.582 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.092) 0:04:17.382 ********* 2021-07-10T09:14:57.653 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-10T09:14:57.655 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.676 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.093) 0:04:17.476 ********* 2021-07-10T09:14:57.746 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-10T09:14:57.748 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.772 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.096) 0:04:17.572 ********* 2021-07-10T09:14:57.843 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.866 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.093) 0:04:17.666 ********* 2021-07-10T09:14:57.938 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-07-10T09:14:57.939 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:57.961 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:57 +0000 (0:00:00.094) 0:04:17.761 ********* 2021-07-10T09:14:58.031 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-10T09:14:58.032 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.054 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.093) 0:04:17.854 ********* 2021-07-10T09:14:58.122 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.145 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.091) 0:04:17.945 ********* 2021-07-10T09:14:58.184 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:14:58.186 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXcRWAOWoZ0musOjxSTFSUsQzHrK7vebzr'], 'uuids': ['9b099c51-2ce5-4a80-9420-eae2433ac630'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.068214", "end": "2021-07-10 09:14:58.152357", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXcRWAOWoZ0musOjxSTFSUsQzHrK7vebzr"], "labels": [], "masters": [], "uuids": ["9b099c51-2ce5-4a80-9420-eae2433ac630"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:14:57.084143", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:14:58.206 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.228 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.083) 0:04:18.028 ********* 2021-07-10T09:14:58.309 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-07-10T09:14:58.311 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.330 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.102) 0:04:18.130 ********* 2021-07-10T09:14:58.399 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-07-10T09:14:58.401 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.422 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.091) 0:04:18.222 ********* 2021-07-10T09:14:58.492 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-07-10T09:14:58.494 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.515 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.092) 0:04:18.315 ********* 2021-07-10T09:14:58.584 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-07-10T09:14:58.586 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.606 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.091) 0:04:18.406 ********* 2021-07-10T09:14:58.676 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.698 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.091) 0:04:18.498 ********* 2021-07-10T09:14:58.766 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-07-10T09:14:58.768 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.788 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.090) 0:04:18.588 ********* 2021-07-10T09:14:58.857 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-07-10T09:14:58.859 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.882 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.093) 0:04:18.682 ********* 2021-07-10T09:14:58.952 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:58.974 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:58 +0000 (0:00:00.091) 0:04:18.774 ********* 2021-07-10T09:14:59.043 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:14:59.044 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-07-10T09:14:59.065 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:59 +0000 (0:00:00.091) 0:04:18.865 ********* 2021-07-10T09:14:59.147 INFO:teuthology.task.ansible.out: 2021-07-10T09:14:59.149 INFO:teuthology.task.ansible.out:TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi062.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.173 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-07-10T09:14:59.183 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.186 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.196 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.198 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.208 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.236 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:59 +0000 (0:00:00.170) 0:04:19.036 ********* 2021-07-10T09:14:59.439 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:14:59.440 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXDuflh4kwWUclfNui3SeMBDYGPz8Uf6aI'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.063090", "end": "2021-07-10 09:14:59.402845", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXDuflh4kwWUclfNui3SeMBDYGPz8Uf6aI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:14:58.339755", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:14:59.505 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-07-10T09:14:59.507 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003336", "end": "2021-07-10 09:14:59.477720", "rc": 0, "start": "2021-07-10 09:14:59.474384", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/virtualenv/bin/teuthology-lock"]} 2021-07-10T09:14:59.531 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:59 +0000 (0:00:00.294) 0:04:19.331 ********* 2021-07-10T09:14:59.611 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-07-10T09:14:59.613 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.636 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:59 +0000 (0:00:00.104) 0:04:19.436 ********* 2021-07-10T09:14:59.705 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-07-10T09:14:59.707 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.728 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:59 +0000 (0:00:00.092) 0:04:19.528 ********* 2021-07-10T09:14:59.798 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-07-10T09:14:59.798 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.821 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:59 +0000 (0:00:00.092) 0:04:19.620 ********* 2021-07-10T09:14:59.892 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-07-10T09:14:59.894 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:14:59.916 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:14:59 +0000 (0:00:00.094) 0:04:19.715 ********* 2021-07-10T09:15:00.686 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:15:00.687 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXCkTDTRVAqeyTkRuXigzpDDQMtOMIiF99'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.058634", "end": "2021-07-10 09:15:00.653056", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXCkTDTRVAqeyTkRuXigzpDDQMtOMIiF99"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:14:59.594422", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:15:00.688 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM 2021-07-10T09:15:00.689 INFO:teuthology.task.ansible.out:0033-9ZM173_Z1W5QCTR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QCTR', 'scsi-35000c50091e03324', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR', 'wwn-0x5000c50091e03324'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QCTR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e03324', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QCTR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QCTR-part1', 'scsi-35000c50091e03324-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR-part1', 'wwn-0x5000c50091e03324-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QCTR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QCTR", "scsi-35000c50091e03324", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR", "wwn-0x5000c50091e03324"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QCTR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QCTR-part1", "scsi-35000c50091e03324-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR-part1", "wwn-0x5000c50091e03324-part 2021-07-10T09:15:00.691 INFO:teuthology.task.ansible.out:1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QCTR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e03324"}}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:01.404 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-07-10T09:15:01.406 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:15:01.426 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:01 +0000 (0:00:01.510) 0:04:21.226 ********* 2021-07-10T09:15:01.516 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-10T09:15:01.518 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930170048512, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227092297, 'block_used': 13002811, 'inode_total': 61054976, 'inode_available': 60984981, 'inode_used': 69995, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227092297, "block_size": 4096, "block_total": 240095108, "block_used": 13002811, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984981, "inode_total": 61054976, "inode_used": 69995, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930170048512, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-07-10T09:15:01.540 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:01 +0000 (0:00:00.114) 0:04:21.340 ********* 2021-07-10T09:15:01.800 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005449", "end": "2021-07-10 09:15:01.772598", "rc": 0, "start": "2021-07-10 09:15:01.767149", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:01.822 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:01 +0000 (0:00:00.282) 0:04:21.622 ********* 2021-07-10T09:15:01.904 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-07-10T09:15:01.906 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930170048512, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227092297, 'block_used': 13002811, 'inode_total': 61054976, 'inode_available': 60984981, 'inode_used': 69995, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227092297, "block_size": 4096, "block_total": 240095108, "block_used": 13002811, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984981, "inode_total": 61054976, "inode_used": 69995, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930170048512, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:01.928 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:01 +0000 (0:00:00.105) 0:04:21.728 ********* 2021-07-10T09:15:01.946 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:15:01.948 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXNsGGDRpje2IrKL3LFiQenflqh5Nf0fwx'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.068689", "end": "2021-07-10 09:15:01.912563", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXNsGGDRpje2IrKL3LFiQenflqh5Nf0fwx"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:00.843874", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:15:01.978 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:01 +0000 (0:00:00.050) 0:04:21.778 ********* 2021-07-10T09:15:01.997 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-07-10T09:15:02.022 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:02 +0000 (0:00:00.043) 0:04:21.822 ********* 2021-07-10T09:15:02.576 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXs0vji8GIQchT4HdQszVzrDJurJ5u4bFB'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.144715", "end": "2021-07-10 09:15:02.548169", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXs0vji8GIQchT4HdQszVzrDJurJ5u4bFB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:02.403454", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141631 s, 814 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141631 s, 814 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:02.795 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-x41AQA-WTmf-Tdu4-h6DN-aVds-xClH-p6Bqd7', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400TR400BGN', 'nvme-nvme.8086-50484654363230343030545234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039483", "end": "2021-07-10 09:15:02.764451", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-x41AQA-WTmf-Tdu4-h6DN-aVds-xClH-p6Bqd7", "nvme-INTEL_SSDPEDMD400G4_PHFT620400TR400BGN", "nvme-nvme.8086-50484654363230343030545234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:02.724968", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0367838 s, 3.1 GB/s", "std 2021-07-10T09:15:02.797 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0367838 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:03.120 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXcRWAOWoZ0musOjxSTFSUsQzHrK7vebzr'], 'uuids': ['9b099c51-2ce5-4a80-9420-eae2433ac630'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.146943", "end": "2021-07-10 09:15:03.092796", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXcRWAOWoZ0musOjxSTFSUsQzHrK7vebzr"], "labels": [], "masters": [], "uuids": ["9b099c51-2ce5-4a80-9420-eae2433ac630"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:02.945853", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144222 s, 800 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144222 s, 800 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:03.435 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXDuflh4kwWUclfNui3SeMBDYGPz8Uf6aI'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.142364", "end": "2021-07-10 09:15:03.407502", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXDuflh4kwWUclfNui3SeMBDYGPz8Uf6aI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:03.265138", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139772 s, 825 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139772 s, 825 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:03.750 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXCkTDTRVAqeyTkRuXigzpDDQMtOMIiF99'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.145413", "end": "2021-07-10 09:15:03.726203", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXCkTDTRVAqeyTkRuXigzpDDQMtOMIiF99"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:03.580790", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142805 s, 808 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142805 s, 808 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:03.762 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QCTR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QCTR', 'scsi-35000c50091e03324', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR', 'wwn-0x5000c50091e03324'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QCTR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e03324', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QCTR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QCTR-part1', 'scsi-35000c50091e03324-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR-part1', 'wwn-0x5000c50091e03324-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QCTR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QCTR", "scsi-35000c50091e03324", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR", "wwn-0x5000c50091e03324"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QCTR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QCTR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-07-10T09:15:03.764 INFO:teuthology.task.ansible.out:Z1W5QCTR-part1", "scsi-35000c50091e03324-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QCTR-part1", "wwn-0x5000c50091e03324-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QCTR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e03324"}}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:03.909 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherZuXDfxb3Cng9s45gIP5dvG3EvQpsIxwQ'], 'uuids': ['3124c616-950a-426c-abff-7681b63e5c76'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.417943", "end": "2021-07-10 09:15:03.871508", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherZuXDfxb3Cng9s45gIP5dvG3EvQpsIxwQ"], "labels": [], "masters": [], "uuids": ["3124c616-950a-426c-abff-7681b63e5c76"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:02.453565", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2021-07-10T09:15:03.911 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2021-07-10T09:15:04.088 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-10T09:15:04.090 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXNsGGDRpje2IrKL3LFiQenflqh5Nf0fwx'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.153443", "end": "2021-07-10 09:15:04.059001", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-KiKE0gAUJUdwtCzYNMoS6D33nGBEmrIXNsGGDRpje2IrKL3LFiQenflqh5Nf0fwx"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:03.905558", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150794 s, 765 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150794 s, 765 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:04.120 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:04 +0000 (0:00:02.097) 0:04:23.919 ********* 2021-07-10T09:15:04.423 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-07-10T09:15:04.424 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.072074", "end": "2021-07-10 09:15:04.351806", "rc": 0, "start": "2021-07-10 09:15:04.279732", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:04.446 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:04 +0000 (0:00:00.326) 0:04:24.246 ********* 2021-07-10T09:15:04.678 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-07-10T09:15:04.680 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049310", "end": "2021-07-10 09:15:04.659308", "rc": 0, "start": "2021-07-10 09:15:04.609998", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:04.701 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:04 +0000 (0:00:00.255) 0:04:24.501 ********* 2021-07-10T09:15:04.781 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-07-10T09:15:04.974 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:04 +0000 (0:00:00.272) 0:04:24.774 ********* 2021-07-10T09:15:05.002 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-07-10T09:15:05.004 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.027 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.052) 0:04:24.827 ********* 2021-07-10T09:15:05.054 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-07-10T09:15:05.056 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.076 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.049) 0:04:24.876 ********* 2021-07-10T09:15:05.104 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-10T09:15:05.106 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.129 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.052) 0:04:24.929 ********* 2021-07-10T09:15:05.147 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:15:05.147 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-j5aT3t-Y3D7-BuKi-Qlmp-xyCe-mSIW-GPfsQW', 'nvme-INTEL_SSDPEDMD400G4_CVFT53440094400BGN', 'nvme-nvme.8086-43564654353334343030393434303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.039680", "end": "2021-07-10 09:15:05.115856", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-j5aT3t-Y3D7-BuKi-Qlmp-xyCe-mSIW-GPfsQW", "nvme-INTEL_SSDPEDMD400G4_CVFT53440094400BGN", "nvme-nvme.8086-43564654353334343030393434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:04.076176", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-07-10T09:15:05.149 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:15:05.159 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-07-10T09:15:05.161 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.182 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.053) 0:04:24.982 ********* 2021-07-10T09:15:05.379 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-07-10T09:15:05.381 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:15:05.403 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.221) 0:04:25.203 ********* 2021-07-10T09:15:05.441 INFO:teuthology.task.ansible.out: 2021-07-10T09:15:05.443 INFO:teuthology.task.ansible.out:TASK [testnode : Set root disk] ************************************************ skipping: [smithi105.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930170159104, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227092324, 'block_used': 13002784, 'inode_total': 61054976, 'inode_available': 60984981, 'inode_used': 69995, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227092324, "block_size": 4096, "block_total": 240095108, "block_used": 13002784, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984981, "inode_total": 61054976, "inode_used": 69995, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930170159104, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.466 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.063) 0:04:25.266 ********* 2021-07-10T09:15:05.494 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-07-10T09:15:05.496 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.516 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.049) 0:04:25.316 ********* 2021-07-10T09:15:05.544 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:05.546 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.566 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.050) 0:04:25.366 ********* 2021-07-10T09:15:05.593 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:05.595 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.615 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.049) 0:04:25.415 ********* 2021-07-10T09:15:05.644 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-07-10T09:15:05.646 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.666 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.050) 0:04:25.466 ********* 2021-07-10T09:15:05.704 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-07-10T09:15:05.705 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.726 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.060) 0:04:25.526 ********* 2021-07-10T09:15:05.753 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-07-10T09:15:05.755 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:05.775 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:05 +0000 (0:00:00.048) 0:04:25.575 ********* 2021-07-10T09:15:06.334 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-07-10T09:15:06.336 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-07-10T09:15:06.358 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:06 +0000 (0:00:00.583) 0:04:26.158 ********* 2021-07-10T09:15:06.428 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:15:06.430 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherCdi5bIEKAdlBuHreIdnBZ1CbPpo906z1'], 'uuids': ['f7088e51-2122-4acd-9f81-d6905e61e537'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.074601", "end": "2021-07-10 09:15:06.391990", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherCdi5bIEKAdlBuHreIdnBZ1CbPpo906z1"], "labels": [], "masters": [], "uuids": ["f7088e51-2122-4acd-9f81-d6905e61e537"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:05.317389", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:15:06.881 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-10T09:15:06.883 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:07.205 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:07.555 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:07.663 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:15:07.666 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherOunLIYi6lfS8mh2Q33GclLlUfAYsPbXM'], 'uuids': ['5a2470d6-ff45-4e8a-a975-266d435cbe56'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.042678", "end": "2021-07-10 09:15:07.634737", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherOunLIYi6lfS8mh2Q33GclLlUfAYsPbXM"], "labels": [], "masters": [], "uuids": ["5a2470d6-ff45-4e8a-a975-266d435cbe56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:06.592059", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:15:07.909 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-10T09:15:07.911 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:08.255 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:08.280 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:08 +0000 (0:00:01.921) 0:04:28.080 ********* 2021-07-10T09:15:08.477 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-07-10T09:15:08.500 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:08 +0000 (0:00:00.219) 0:04:28.300 ********* 2021-07-10T09:15:08.715 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-10T09:15:08.717 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:08.896 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:08.949 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:15:08.951 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherTAseT6I409ZgePJAbuV26LPd7HwUknYX'], 'uuids': ['d34e744d-c021-4d96-bf7e-8dbde284a5ad'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.066849", "end": "2021-07-10 09:15:08.914901", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherTAseT6I409ZgePJAbuV26LPd7HwUknYX"], "labels": [], "masters": [], "uuids": ["d34e744d-c021-4d96-bf7e-8dbde284a5ad"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:07.848052", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi062.front.sepia.ceph.com] => 2021-07-10T09:15:08.953 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W5SV', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4W5SV', 'scsi-35000c50087b99812', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV', 'wwn-0x5000c50087b99812'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4W5SV', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50087b99812', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W5SV-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4W5SV-part1', 'scsi-35000c50087b99812-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV-part1', 'wwn-0x5000c50087b99812-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W5SV", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4W5SV", "scsi-35000c50087b99812", "scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV", "wwn-0x5000c50087b99812"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W5SV-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4W5SV-part1", "scsi-35000c50087b99812-part1 2021-07-10T09:15:08.954 INFO:teuthology.task.ansible.out:", "scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV-part1", "wwn-0x5000c50087b99812-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W5SV", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b99812"}}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:09.067 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-10T09:15:09.069 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:09.237 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:09.248 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:09.274 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:09 +0000 (0:00:00.774) 0:04:29.074 ********* 2021-07-10T09:15:09.302 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-07-10T09:15:09.304 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:09.328 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:09 +0000 (0:00:00.053) 0:04:29.128 ********* 2021-07-10T09:15:09.356 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-07-10T09:15:09.358 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:09.380 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:09 +0000 (0:00:00.052) 0:04:29.180 ********* 2021-07-10T09:15:09.597 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:15:09.620 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:09 +0000 (0:00:00.239) 0:04:29.420 ********* 2021-07-10T09:15:09.658 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-07-10T09:15:09.660 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-07-10T09:15:09.683 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:09 +0000 (0:00:00.062) 0:04:29.483 ********* 2021-07-10T09:15:10.111 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-07-10T09:15:10.112 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true} 2021-07-10T09:15:10.135 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.452) 0:04:29.935 ********* 2021-07-10T09:15:10.217 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-10T09:15:10.219 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherDXblsc8PyY5I0Bx3RD3YZgVMOmQMIKu0'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.065019", "end": "2021-07-10 09:15:10.186442", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherDXblsc8PyY5I0Bx3RD3YZgVMOmQMIKu0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:09.121423", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-10T09:15:10.247 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.111) 0:04:30.047 ********* 2021-07-10T09:15:10.591 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-07-10T09:15:10.593 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-07-10T09:15:10.614 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.367) 0:04:30.414 ********* 2021-07-10T09:15:10.633 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-10T09:15:10.634 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherZuXDfxb3Cng9s45gIP5dvG3EvQpsIxwQ'], 'uuids': ['3124c616-950a-426c-abff-7681b63e5c76'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.151859", "end": "2021-07-10 09:15:10.604975", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherZuXDfxb3Cng9s45gIP5dvG3EvQpsIxwQ"], "labels": [], "masters": [], "uuids": ["3124c616-950a-426c-abff-7681b63e5c76"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:10.453116", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148808 s, 775 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148808 s, 775 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:10.646 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-07-10T09:15:10.647 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:10.670 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.055) 0:04:30.470 ********* 2021-07-10T09:15:10.698 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:10.722 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.052) 0:04:30.522 ********* 2021-07-10T09:15:10.753 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-10T09:15:10.754 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:10.777 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.054) 0:04:30.577 ********* 2021-07-10T09:15:10.805 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-10T09:15:10.807 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:10.832 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.055) 0:04:30.632 ********* 2021-07-10T09:15:10.861 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-10T09:15:10.863 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-j5aT3t-Y3D7-BuKi-Qlmp-xyCe-mSIW-GPfsQW', 'nvme-INTEL_SSDPEDMD400G4_CVFT53440094400BGN', 'nvme-nvme.8086-43564654353334343030393434303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039921", "end": "2021-07-10 09:15:10.828653", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-j5aT3t-Y3D7-BuKi-Qlmp-xyCe-mSIW-GPfsQW", "nvme-INTEL_SSDPEDMD400G4_CVFT53440094400BGN", "nvme-nvme.8086-43564654353334343030393434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:10.788732", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0366229 s, 3.1 GB/s", "std 2021-07-10T09:15:10.865 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0366229 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:10.890 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.057) 0:04:30.690 ********* 2021-07-10T09:15:10.918 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-07-10T09:15:10.920 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:10.941 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.051) 0:04:30.741 ********* 2021-07-10T09:15:10.969 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-07-10T09:15:10.971 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:10.993 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:10 +0000 (0:00:00.052) 0:04:30.793 ********* 2021-07-10T09:15:11.020 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-07-10T09:15:11.022 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.042 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.049) 0:04:30.842 ********* 2021-07-10T09:15:11.069 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-07-10T09:15:11.071 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.091 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.048) 0:04:30.891 ********* 2021-07-10T09:15:11.118 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-07-10T09:15:11.120 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.140 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.049) 0:04:30.940 ********* 2021-07-10T09:15:11.178 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-07-10T09:15:11.179 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.200 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.059) 0:04:31.000 ********* 2021-07-10T09:15:11.207 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-10T09:15:11.208 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherCdi5bIEKAdlBuHreIdnBZ1CbPpo906z1'], 'uuids': ['f7088e51-2122-4acd-9f81-d6905e61e537'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.145527", "end": "2021-07-10 09:15:11.163733", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherCdi5bIEKAdlBuHreIdnBZ1CbPpo906z1"], "labels": [], "masters": [], "uuids": ["f7088e51-2122-4acd-9f81-d6905e61e537"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:11.018206", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142788 s, 808 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142788 s, 808 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:11.230 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.252 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.051) 0:04:31.052 ********* 2021-07-10T09:15:11.279 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-07-10T09:15:11.281 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.301 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.049) 0:04:31.101 ********* 2021-07-10T09:15:11.328 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.350 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.049) 0:04:31.150 ********* 2021-07-10T09:15:11.378 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.403 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.053) 0:04:31.203 ********* 2021-07-10T09:15:11.531 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-10T09:15:11.533 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherOunLIYi6lfS8mh2Q33GclLlUfAYsPbXM'], 'uuids': ['5a2470d6-ff45-4e8a-a975-266d435cbe56'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.145912", "end": "2021-07-10 09:15:11.493010", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherOunLIYi6lfS8mh2Q33GclLlUfAYsPbXM"], "labels": [], "masters": [], "uuids": ["5a2470d6-ff45-4e8a-a975-266d435cbe56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:11.347098", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143198 s, 805 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143198 s, 805 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:11.856 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherTAseT6I409ZgePJAbuV26LPd7HwUknYX'], 'uuids': ['d34e744d-c021-4d96-bf7e-8dbde284a5ad'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.155977", "end": "2021-07-10 09:15:11.832646", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherTAseT6I409ZgePJAbuV26LPd7HwUknYX"], "labels": [], "masters": [], "uuids": ["d34e744d-c021-4d96-bf7e-8dbde284a5ad"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:11.676669", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153153 s, 753 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153153 s, 753 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:11.868 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W5SV', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4W5SV', 'scsi-35000c50087b99812', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV', 'wwn-0x5000c50087b99812'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4W5SV', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50087b99812', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W5SV-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4W5SV-part1', 'scsi-35000c50087b99812-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV-part1', 'wwn-0x5000c50087b99812-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W5SV", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4W5SV", "scsi-35000c50087b99812", "scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV", "wwn-0x5000c50087b99812"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W5SV-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4W5SV-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-07-10T09:15:11.870 INFO:teuthology.task.ansible.out:Z1W4W5SV-part1", "scsi-35000c50087b99812-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4W5SV-part1", "wwn-0x5000c50087b99812-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W5SV", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b99812"}}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:11.891 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-07-10T09:15:11.893 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-07-10T09:15:11.915 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:11 +0000 (0:00:00.511) 0:04:31.715 ********* 2021-07-10T09:15:12.205 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-10T09:15:12.207 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherDXblsc8PyY5I0Bx3RD3YZgVMOmQMIKu0'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.145338", "end": "2021-07-10 09:15:12.172984", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-uk1ya8uZx4kkcXN0BRnn5c2awG2pgherDXblsc8PyY5I0Bx3RD3YZgVMOmQMIKu0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-10 09:15:12.027646", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142521 s, 809 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142521 s, 809 MB/s"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:12.238 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.323) 0:04:32.038 ********* 2021-07-10T09:15:12.247 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-07-10T09:15:12.249 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-10 09:10:13 UTC", "ActiveEnterTimestampMonotonic": "12773217", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice sntp.service ntpd.service systemd-tmpfiles-setup.service sysinit.target systemd-journald.socket tmp.mount ntpdate.service -.mount basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-10 09:10:12 UTC", "AssertTimestampMonotonic": "11562514", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-10 09:10:12 UTC", "ConditionTimestampMonotonic": "11562475", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.tar 2021-07-10T09:15:12.251 INFO:teuthology.task.ansible.out:get ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "878", "ExecMainStartTimestamp": "Sat 2021-07-10 09:10:13 UTC", "ExecMainStartTimestampMonotonic": "12721345", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-07-10 09:10:12 UTC", "InactiveExitTimestampMonotonic": "12073835", "InvocationID": "36c7f528b97f45dfa31810529e24cb8b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinit 2021-07-10T09:15:12.252 INFO:teuthology.task.ansible.out:y", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "878", "MemoryAccounting": "yes", "MemoryCurrent": "1589248", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target - 2021-07-10T09:15:12.254 INFO:teuthology.task.ansible.out:.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-10 09:10:13 UTC", "StateChangeTimestampMonotonic": "12773217", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-10 09:10:13 UTC", "WatchdogTimestampMonotonic": "12721369", "WatchdogUSec": "0"}} 2021-07-10T09:15:12.276 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.037) 0:04:32.076 ********* 2021-07-10T09:15:12.304 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:12.306 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-07-10T09:15:12.326 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.050) 0:04:32.126 ********* 2021-07-10T09:15:12.368 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-07-10T09:15:12.388 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.061) 0:04:32.188 ********* 2021-07-10T09:15:12.415 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-07-10T09:15:12.417 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:12.438 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.050) 0:04:32.238 ********* 2021-07-10T09:15:12.465 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:12.487 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.049) 0:04:32.287 ********* 2021-07-10T09:15:12.494 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-07-10T09:15:12.495 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.057434", "end": "2021-07-10 09:15:12.464771", "rc": 0, "start": "2021-07-10 09:15:12.407337", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:12.518 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.031) 0:04:32.318 ********* 2021-07-10T09:15:12.525 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-07-10T09:15:12.527 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:12.549 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.031) 0:04:32.349 ********* 2021-07-10T09:15:12.754 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-07-10T09:15:12.756 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} TASK [testnode : Check for physical volumes] *********************************** changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.050811", "end": "2021-07-10 09:15:12.735750", "rc": 0, "start": "2021-07-10 09:15:12.684939", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:12.779 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.229) 0:04:32.579 ********* 2021-07-10T09:15:12.800 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.020) 0:04:32.600 ********* 2021-07-10T09:15:12.808 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-07-10T09:15:12.841 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.041) 0:04:32.641 ********* 2021-07-10T09:15:12.905 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-07-10T09:15:12.906 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:12.938 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.096) 0:04:32.738 ********* 2021-07-10T09:15:12.966 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-07-10T09:15:12.967 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:12.993 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:12 +0000 (0:00:00.054) 0:04:32.793 ********* 2021-07-10T09:15:13.028 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-10T09:15:13.030 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.054 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.061) 0:04:32.854 ********* 2021-07-10T09:15:13.102 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-07-10T09:15:13.104 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.124 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.069) 0:04:32.924 ********* 2021-07-10T09:15:13.329 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi062.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-10T09:15:13.352 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.227) 0:04:33.152 ********* 2021-07-10T09:15:13.359 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-07-10T09:15:13.359 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-07-10T09:15:13.390 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.038) 0:04:33.190 ********* 2021-07-10T09:15:13.419 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-10T09:15:13.421 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930170048512, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227092297, 'block_used': 13002811, 'inode_total': 61054976, 'inode_available': 60984981, 'inode_used': 69995, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227092297, "block_size": 4096, "block_total": 240095108, "block_used": 13002811, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984981, "inode_total": 61054976, "inode_used": 69995, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930170048512, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.444 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.053) 0:04:33.244 ********* 2021-07-10T09:15:13.472 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-07-10T09:15:13.474 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.496 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.051) 0:04:33.296 ********* 2021-07-10T09:15:13.523 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:13.525 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.547 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.051) 0:04:33.347 ********* 2021-07-10T09:15:13.574 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:13.576 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.597 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.050) 0:04:33.397 ********* 2021-07-10T09:15:13.625 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-07-10T09:15:13.627 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.648 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.050) 0:04:33.448 ********* 2021-07-10T09:15:13.675 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-07-10T09:15:13.677 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.698 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.050) 0:04:33.498 ********* 2021-07-10T09:15:13.725 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-07-10T09:15:13.727 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:13.748 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:13 +0000 (0:00:00.049) 0:04:33.548 ********* 2021-07-10T09:15:14.177 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-07-10T09:15:14.199 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:14 +0000 (0:00:00.451) 0:04:33.999 ********* 2021-07-10T09:15:14.608 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-07-10T09:15:14.610 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-07-10T09:15:14.632 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:14 +0000 (0:00:00.432) 0:04:34.432 ********* 2021-07-10T09:15:14.640 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-10T09:15:14.642 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:14.663 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:14.685 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:14 +0000 (0:00:00.053) 0:04:34.485 ********* 2021-07-10T09:15:14.714 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-07-10T09:15:14.715 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-07-10T09:15:14.748 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:14 +0000 (0:00:00.062) 0:04:34.548 ********* 2021-07-10T09:15:14.934 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-07-10T09:15:14.936 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-07-10T09:15:14.960 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:14 +0000 (0:00:00.212) 0:04:34.760 ********* 2021-07-10T09:15:14.988 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-07-10T09:15:14.990 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.010 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.049) 0:04:34.810 ********* 2021-07-10T09:15:15.017 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-10T09:15:15.018 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:15.040 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-07-10T09:15:15.042 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-07-10T09:15:15.062 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.051) 0:04:34.862 ********* 2021-07-10T09:15:15.099 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-07-10T09:15:15.101 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.121 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.059) 0:04:34.921 ********* 2021-07-10T09:15:15.308 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-07-10T09:15:15.310 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-07-10T09:15:15.330 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.208) 0:04:35.130 ********* 2021-07-10T09:15:15.388 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-10T09:15:15.389 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:15.525 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-07-10T09:15:15.526 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-07-10T09:15:15.548 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.218) 0:04:35.348 ********* 2021-07-10T09:15:15.575 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-07-10T09:15:15.577 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.598 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.049) 0:04:35.398 ********* 2021-07-10T09:15:15.636 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-07-10T09:15:15.637 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.648 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.649 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.671 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.073) 0:04:35.471 ********* 2021-07-10T09:15:15.709 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-07-10T09:15:15.711 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi105.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.721 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.723 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.733 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-10T09:15:15.734 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi105.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.757 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.085) 0:04:35.557 ********* 2021-07-10T09:15:15.784 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:15.786 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.809 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.051) 0:04:35.609 ********* 2021-07-10T09:15:15.836 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-07-10T09:15:15.838 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.858 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.049) 0:04:35.658 ********* 2021-07-10T09:15:15.885 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-07-10T09:15:15.887 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.909 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.051) 0:04:35.709 ********* 2021-07-10T09:15:15.936 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-07-10T09:15:15.938 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:15.958 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:15 +0000 (0:00:00.049) 0:04:35.759 ********* 2021-07-10T09:15:15.986 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:16.011 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:16 +0000 (0:00:00.052) 0:04:35.811 ********* 2021-07-10T09:15:16.040 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-10T09:15:16.041 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:16.065 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:16 +0000 (0:00:00.054) 0:04:35.865 ********* 2021-07-10T09:15:16.082 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-10T09:15:16.083 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-10T09:15:16.111 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:16 +0000 (0:00:00.045) 0:04:35.911 ********* 2021-07-10T09:15:16.118 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-07-10T09:15:16.120 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:16.146 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:16 +0000 (0:00:00.034) 0:04:35.946 ********* 2021-07-10T09:15:16.173 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-10T09:15:16.174 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:16.196 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:16 +0000 (0:00:00.050) 0:04:35.996 ********* 2021-07-10T09:15:16.317 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-10T09:15:16.319 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-07-10T09:15:16.341 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:16 +0000 (0:00:00.145) 0:04:36.141 ********* 2021-07-10T09:15:16.416 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-07-10T09:15:16.417 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005065", "end": "2021-07-10 09:15:16.375885", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-07-10 09:15:16.370820", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:16.580 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:16.592 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-07-10T09:15:16.594 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002937", "end": "2021-07-10 09:15:16.560541", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-07-10 09:15:16.557604", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:16.762 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-10T09:15:16.764 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:16.764 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-07-10T09:15:16.764 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002909", "end": "2021-07-10 09:15:16.736932", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-07-10 09:15:16.734023", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:16.937 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003217", "end": "2021-07-10 09:15:16.908160", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-07-10 09:15:16.904943", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:16.967 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:16 +0000 (0:00:00.625) 0:04:36.767 ********* 2021-07-10T09:15:16.977 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-10T09:15:16.979 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:16.990 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-07-10T09:15:17.013 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:17 +0000 (0:00:00.045) 0:04:36.813 ********* 2021-07-10T09:15:17.041 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-07-10T09:15:17.043 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:17.127 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-10T09:15:17.129 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-10T09:15:17.139 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:17.163 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:17 +0000 (0:00:00.149) 0:04:36.963 ********* 2021-07-10T09:15:17.190 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:17.212 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:17 +0000 (0:00:00.048) 0:04:37.011 ********* 2021-07-10T09:15:17.239 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-07-10T09:15:17.241 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:17.261 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:17 +0000 (0:00:00.050) 0:04:37.062 ********* 2021-07-10T09:15:17.263 INFO:teuthology.task.ansible.out: 2021-07-10T09:15:17.469 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-07-10T09:15:17.470 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:15:17.492 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:17 +0000 (0:00:00.230) 0:04:37.292 ********* 2021-07-10T09:15:17.530 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-07-10T09:15:17.531 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-07-10T09:15:17.552 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:17 +0000 (0:00:00.060) 0:04:37.352 ********* 2021-07-10T09:15:17.842 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-07-10T09:15:17.844 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true} 2021-07-10T09:15:17.865 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:17 +0000 (0:00:00.312) 0:04:37.665 ********* 2021-07-10T09:15:18.193 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-07-10T09:15:18.216 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.348) 0:04:38.014 ********* 2021-07-10T09:15:18.242 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-07-10T09:15:18.244 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.266 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.051) 0:04:38.066 ********* 2021-07-10T09:15:18.295 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.319 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.052) 0:04:38.119 ********* 2021-07-10T09:15:18.346 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-10T09:15:18.349 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.369 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.169 ********* 2021-07-10T09:15:18.397 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-10T09:15:18.398 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.424 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.055) 0:04:38.224 ********* 2021-07-10T09:15:18.452 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-07-10T09:15:18.454 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.476 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.051) 0:04:38.275 ********* 2021-07-10T09:15:18.503 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.526 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.326 ********* 2021-07-10T09:15:18.554 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-07-10T09:15:18.556 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.577 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.051) 0:04:38.377 ********* 2021-07-10T09:15:18.604 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-07-10T09:15:18.606 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.627 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.427 ********* 2021-07-10T09:15:18.655 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-07-10T09:15:18.657 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.677 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.478 ********* 2021-07-10T09:15:18.706 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.728 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.528 ********* 2021-07-10T09:15:18.756 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-07-10T09:15:18.758 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.779 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.579 ********* 2021-07-10T09:15:18.807 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-07-10T09:15:18.808 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.830 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.630 ********* 2021-07-10T09:15:18.858 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.881 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.680 ********* 2021-07-10T09:15:18.909 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-07-10T09:15:18.910 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.931 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.050) 0:04:38.731 ********* 2021-07-10T09:15:18.959 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-07-10T09:15:18.961 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:18.982 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:18 +0000 (0:00:00.051) 0:04:38.782 ********* 2021-07-10T09:15:19.483 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-07-10T09:15:19.505 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:19 +0000 (0:00:00.522) 0:04:39.305 ********* 2021-07-10T09:15:19.823 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-07-10 09:10:12 UTC", "ActiveEnterTimestampMonotonic": "12600487", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket systemd-tmpfiles-setup.service basic.target -.mount tmp.mount system.slice ntpd.service ntpdate.service sntp.service sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-07-10 09:10:10 UTC", "AssertTimestampMonotonic": "11334516", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-07-10 09:10:10 UTC", "ConditionTimestampMono 2021-07-10T09:15:19.825 INFO:teuthology.task.ansible.out:tonic": "11334472", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "875", "ExecMainStartTimestamp": "Sat 2021-07-10 09:10:12 UTC", "ExecMainStartTimestampMonotonic": "12529319", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-07-10 09:10:11 UTC", "InactiveExitTimestampMonotonic": "11460071", "InvocationID": "0ec7be026be54df78822e34f63471dad", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": 2021-07-10T09:15:19.827 INFO:teuthology.task.ansible.out: "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "875", "MemoryAccounting": "yes", "MemoryCurrent": "1597440", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "R 2021-07-10T09:15:19.829 INFO:teuthology.task.ansible.out:emainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-07-10 09:10:12 UTC", "StateChangeTimestampMonotonic": "12600487", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-07-10 09:10:12 UTC", "WatchdogTimestampMonotonic": "12529579", "WatchdogUSec": "0"}} 2021-07-10T09:15:19.849 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:19 +0000 (0:00:00.343) 0:04:39.649 ********* 2021-07-10T09:15:19.877 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:19.878 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-07-10T09:15:19.901 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:19 +0000 (0:00:00.052) 0:04:39.701 ********* 2021-07-10T09:15:19.939 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:19.940 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-07-10T09:15:19.965 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:19 +0000 (0:00:00.063) 0:04:39.765 ********* 2021-07-10T09:15:19.995 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:20.018 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:20 +0000 (0:00:00.053) 0:04:39.818 ********* 2021-07-10T09:15:20.047 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-07-10T09:15:20.049 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:20.071 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:20 +0000 (0:00:00.052) 0:04:39.870 ********* 2021-07-10T09:15:20.100 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-07-10T09:15:20.101 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:20.123 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:20 +0000 (0:00:00.052) 0:04:39.923 ********* 2021-07-10T09:15:20.332 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-07-10T09:15:20.333 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-07-10T09:15:20.355 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:20 +0000 (0:00:00.232) 0:04:40.155 ********* 2021-07-10T09:15:20.863 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-07-10T09:15:20.865 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-07-10T09:15:20.886 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:20 +0000 (0:00:00.530) 0:04:40.686 ********* 2021-07-10T09:15:21.905 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-07-10T09:15:21.927 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:21 +0000 (0:00:01.041) 0:04:41.727 ********* 2021-07-10T09:15:21.954 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-07-10T09:15:21.956 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:21.977 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:21 +0000 (0:00:00.049) 0:04:41.777 ********* 2021-07-10T09:15:22.007 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-07-10T09:15:22.009 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-07-10T09:15:22.034 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.056) 0:04:41.834 ********* 2021-07-10T09:15:22.233 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-07-10T09:15:22.235 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-07-10T09:15:22.258 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.224) 0:04:42.058 ********* 2021-07-10T09:15:22.287 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:22.309 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.051) 0:04:42.109 ********* 2021-07-10T09:15:22.343 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-07-10T09:15:22.345 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-07-10T09:15:22.367 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.056) 0:04:42.166 ********* 2021-07-10T09:15:22.396 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:22.418 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.051) 0:04:42.218 ********* 2021-07-10T09:15:22.627 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi062.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-07-10T09:15:22.649 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.231) 0:04:42.449 ********* 2021-07-10T09:15:22.850 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi062.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-07-10T09:15:22.878 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.227) 0:04:42.677 ********* 2021-07-10T09:15:22.905 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-07-10T09:15:22.906 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:22.928 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:22 +0000 (0:00:00.051) 0:04:42.728 ********* 2021-07-10T09:15:22.968 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi062.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-07-10T09:15:22.978 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi062.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.006 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.076) 0:04:42.804 ********* 2021-07-10T09:15:23.057 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi062.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.060 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.070 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.079 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.081 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.105 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.098) 0:04:42.905 ********* 2021-07-10T09:15:23.132 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-10T09:15:23.133 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.156 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.051) 0:04:42.956 ********* 2021-07-10T09:15:23.157 INFO:teuthology.task.ansible.out: 2021-07-10T09:15:23.185 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-07-10T09:15:23.187 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.209 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.052) 0:04:43.009 ********* 2021-07-10T09:15:23.240 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.264 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.055) 0:04:43.064 ********* 2021-07-10T09:15:23.295 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.318 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.054) 0:04:43.118 ********* 2021-07-10T09:15:23.347 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-07-10T09:15:23.348 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.373 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.055) 0:04:43.173 ********* 2021-07-10T09:15:23.401 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-10T09:15:23.403 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.428 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.054) 0:04:43.228 ********* 2021-07-10T09:15:23.456 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-07-10T09:15:23.458 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.483 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.054) 0:04:43.282 ********* 2021-07-10T09:15:23.511 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-07-10T09:15:23.513 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:23.534 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:23 +0000 (0:00:00.050) 0:04:43.333 ********* 2021-07-10T09:15:24.789 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-07-10T09:15:24.791 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006697", "end": "2021-07-10 09:15:24.748332", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-07-10 09:15:23.741635", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:25.996 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:01.004350", "end": "2021-07-10 09:15:25.953785", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-07-10 09:15:24.949435", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:27.198 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:01.004283", "end": "2021-07-10 09:15:27.156341", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-07-10 09:15:26.152058", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:27.387 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003445", "end": "2021-07-10 09:15:27.352984", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-07-10 09:15:27.349539", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:27.415 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:27 +0000 (0:00:03.881) 0:04:47.215 ********* 2021-07-10T09:15:27.445 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-07-10T09:15:27.467 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:27 +0000 (0:00:00.051) 0:04:47.267 ********* 2021-07-10T09:15:27.494 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-07-10T09:15:27.496 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:27.563 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-07-10T09:15:27.584 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:27 +0000 (0:00:00.116) 0:04:47.384 ********* 2021-07-10T09:15:27.611 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:27 +0000 (0:00:00.026) 0:04:47.411 ********* 2021-07-10T09:15:28.240 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi062.front.sepia.ceph.com] 2021-07-10T09:15:28.273 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] 2021-07-10T09:15:28.300 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.688) 0:04:48.100 ********* 2021-07-10T09:15:28.340 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.040) 0:04:48.140 ********* 2021-07-10T09:15:28.347 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-10T09:15:28.348 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:28.371 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.030) 0:04:48.171 ********* 2021-07-10T09:15:28.378 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:28.402 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.030) 0:04:48.202 ********* 2021-07-10T09:15:28.412 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-10T09:15:28.414 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-07-10T09:15:28.443 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.040) 0:04:48.243 ********* 2021-07-10T09:15:28.453 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-07-10T09:15:28.477 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.034) 0:04:48.277 ********* 2021-07-10T09:15:28.486 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-07-10T09:15:28.488 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-07-10T09:15:28.514 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.036) 0:04:48.314 ********* 2021-07-10T09:15:28.535 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-07-10T09:15:28.560 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:28 +0000 (0:00:00.045) 0:04:48.360 ********* 2021-07-10T09:15:29.963 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-07-10T09:15:29.965 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:15:29.986 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:29 +0000 (0:00:01.425) 0:04:49.786 ********* 2021-07-10T09:15:29.994 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:15:30.018 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:30 +0000 (0:00:00.032) 0:04:49.818 ********* 2021-07-10T09:15:30.025 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-10T09:15:30.026 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-07-10T09:15:30.049 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:30 +0000 (0:00:00.030) 0:04:49.849 ********* 2021-07-10T09:15:30.057 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-07-10T09:15:30.080 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:30 +0000 (0:00:00.031) 0:04:49.880 ********* 2021-07-10T09:15:30.088 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-10T09:15:30.090 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:30.112 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:30 +0000 (0:00:00.031) 0:04:49.912 ********* 2021-07-10T09:15:30.119 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:30.144 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:30 +0000 (0:00:00.031) 0:04:49.944 ********* 2021-07-10T09:15:30.329 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-07-10T09:15:30.331 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:15:30.362 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:15:30.510 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:15:30.538 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:30 +0000 (0:00:00.394) 0:04:50.338 ********* 2021-07-10T09:15:30.563 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-07-10T09:15:30.591 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:30 +0000 (0:00:00.052) 0:04:50.391 ********* 2021-07-10T09:15:31.088 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-07-10T09:15:31.090 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908530.6199627-24797-105109247168470/source", "state": "file", "uid": 0} 2021-07-10T09:15:31.132 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908530.658173-24801-215906382185416/source", "state": "file", "uid": 0} 2021-07-10T09:15:31.573 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908531.1199281-24797-53599698083389/source", "state": "file", "uid": 0} 2021-07-10T09:15:31.598 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:31 +0000 (0:00:01.007) 0:04:51.398 ********* 2021-07-10T09:15:31.648 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1625908531.1709228-24801-15874740927632/source", "state": "file", "uid": 0} 2021-07-10T09:15:31.676 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:31 +0000 (0:00:00.076) 0:04:51.475 ********* 2021-07-10T09:15:32.951 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-07-10T09:15:32.953 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:15:32.976 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:32 +0000 (0:00:01.301) 0:04:52.776 ********* 2021-07-10T09:15:33.255 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-07-10T09:15:33.278 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:33 +0000 (0:00:00.302) 0:04:53.078 ********* 2021-07-10T09:15:36.106 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-07-10T09:15:36.108 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-eus1cz7m-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-eus1cz7m-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-07-10T09:15:36.110 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-07-10T09:15:36.129 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:36 +0000 (0:00:02.851) 0:04:55.929 ********* 2021-07-10T09:15:36.158 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-07-10T09:15:36.160 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:36.183 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:36 +0000 (0:00:00.053) 0:04:55.983 ********* 2021-07-10T09:15:36.222 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:36.246 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:36 +0000 (0:00:00.061) 0:04:56.045 ********* 2021-07-10T09:15:36.598 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-07-10T09:15:36.600 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.158511", "end": "2021-07-10 09:15:36.576234", "rc": 0, "start": "2021-07-10 09:15:36.417723", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:36.621 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:36 +0000 (0:00:00.375) 0:04:56.421 ********* 2021-07-10T09:15:36.650 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-07-10T09:15:36.652 INFO:teuthology.task.ansible.out:skipping: [smithi105.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:36.734 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-07-10T09:15:36.735 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-_sj3y1ss-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-_sj3y1ss-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-07-10T09:15:36.737 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-07-10T09:15:36.755 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:36 +0000 (0:00:00.134) 0:04:56.555 ********* 2021-07-10T09:15:36.782 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-07-10T09:15:36.784 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:36.804 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:36 +0000 (0:00:00.048) 0:04:56.604 ********* 2021-07-10T09:15:36.832 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-07-10T09:15:36.834 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:36.857 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:36 +0000 (0:00:00.052) 0:04:56.657 ********* 2021-07-10T09:15:37.231 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-07-10T09:15:37.233 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.168815", "end": "2021-07-10 09:15:37.204009", "rc": 0, "start": "2021-07-10 09:15:37.035194", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-10T09:15:37.254 INFO:teuthology.task.ansible.out:Saturday 10 July 2021 09:15:37 +0000 (0:00:00.396) 0:04:57.054 ********* 2021-07-10T09:15:37.281 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-07-10T09:15:37.283 INFO:teuthology.task.ansible.out:skipping: [smithi062.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-10T09:15:37.350 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-07-10T09:15:37.351 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-07-10T09:15:37.352 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-07-10T09:15:37.354 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-10T09:15:37.365 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 10 July 2021 09:15:37 +0000 (0:00:00.111) 0:04:57.165 ********* 2021-07-10T09:15:38.022 INFO:teuthology.task.ansible.out:ok: [smithi062.front.sepia.ceph.com] 2021-07-10T09:15:38.041 INFO:teuthology.task.ansible.out:ok: [smithi105.front.sepia.ceph.com] 2021-07-10T09:15:38.057 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Saturday 10 July 2021 09:15:38 +0000 (0:00:00.691) 0:04:57.857 ********* 2021-07-10T09:15:38.263 INFO:teuthology.task.ansible.out:changed: [smithi062.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-10T09:15:38.275 INFO:teuthology.task.ansible.out:changed: [smithi105.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-07-10T09:15:38.277 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-07-10T09:15:38.279 INFO:teuthology.task.ansible.out:smithi062.front.sepia.ceph.com : ok=134 changed=46 unreachable=0 failed=0 skipped=185 rescued=0 ignored=3 smithi105.front.sepia.ceph.com : ok=134 changed=46 unreachable=0 failed=0 skipped=185 rescued=0 ignored=3 Saturday 10 July 2021 09:15:38 +0000 (0:00:00.220) 0:04:58.078 ********* =============================================================================== Create all admin users with sudo access. ------------------------------- 49.84s common : nrpe - Ensure SELinux policy is up to date -------------------- 34.88s users : Update authorized_keys using the keys repo --------------------- 29.39s common : nrpe - Remove SELinux policy package -------------------------- 11.21s common : nrpe - Load SELinux policy package ---------------------------- 10.01s Remove revoked users ---------------------------------------------------- 5.73s testnode : List any leftover Ceph artifacts from previous jobs ---------- 3.88s container-host : Install registries-conf-ctl ---------------------------- 2.85s testnode : Ensure ceph dependency packages are not present. ------------- 2.62s testnode : Ensure ceph packages are not present. ------------------------ 2.48s testnode : Zap all non-root disks --------------------------------------- 2.10s users : Update authorized_keys for each user with literal keys ---------- 1.85s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.56s testnode : Configure local mirrorlists ---------------------------------- 1.56s Gathering Facts --------------------------------------------------------- 1.56s Gathering Facts --------------------------------------------------------- 1.54s common : nrpe - Install SELinux dependencies ---------------------------- 1.52s testnode : Make sure rpm dependencies are installed --------------------- 1.51s container-host : Install container packages ----------------------------- 1.43s container-host : Ensure git is installed -------- 2021-07-10T09:15:38.281 INFO:teuthology.task.ansible.out:------------------------ 1.30s 2021-07-10T09:15:38.741 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2021-07-10T09:15:38.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-10T09:15:38.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:15:38.826 DEBUG:teuthology.orchestra.run.smithi062:> true 2021-07-10T09:15:38.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-10T09:15:38.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-10T09:15:38.964 DEBUG:teuthology.orchestra.run.smithi105:> true 2021-07-10T09:15:39.031 INFO:teuthology.run_tasks:Running task clock... 2021-07-10T09:15:39.040 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-07-10T09:15:39.041 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-07-10T09:15:39.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-07-10T09:15:39.075 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-07-10T09:15:39.091 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-07-10T09:15:39.130 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2021-07-10T09:15:39.142 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2021-07-10T09:15:39.156 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-07-10T09:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-07-10T09:15:39.234 INFO:teuthology.orchestra.run.smithi062.stderr:bash: ntpq: command not found 2021-07-10T09:15:39.237 INFO:teuthology.orchestra.run.smithi062.stdout:210 Number of sources = 4 2021-07-10T09:15:39.237 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-07-10T09:15:39.237 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2021-07-10T09:15:39.238 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.238 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.238 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.238 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.239 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-07-10T09:15:39.240 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-07-10T09:15:39.272 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-07-10T09:15:39.287 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-07-10T09:15:39.324 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2021-07-10T09:15:39.334 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2021-07-10T09:15:39.348 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-07-10T09:15:39.362 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-07-10T09:15:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:bash: ntpq: command not found 2021-07-10T09:15:39.444 INFO:teuthology.orchestra.run.smithi105.stdout:210 Number of sources = 4 2021-07-10T09:15:39.444 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-07-10T09:15:39.444 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2021-07-10T09:15:39.444 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.445 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.445 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.445 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-07-10T09:15:39.446 INFO:teuthology.run_tasks:Running task install... 2021-07-10T09:15:39.457 DEBUG:teuthology.task.install:project ceph 2021-07-10T09:15:39.457 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}} 2021-07-10T09:15:39.458 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'} 2021-07-10T09:15:39.459 INFO:teuthology.task.install:Using flavor: basic 2021-07-10T09:15:39.471 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-07-10T09:15:39.472 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2021-07-10T09:15:39.473 DEBUG:teuthology.orchestra.run.smithi062:> sudo lsb_release -is 2021-07-10T09:15:39.511 INFO:teuthology.orchestra.run.smithi062.stdout:CentOSStream 2021-07-10T09:15:39.512 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-07-10T09:15:39.512 DEBUG:teuthology.orchestra.run.smithi105:> sudo lsb_release -is 2021-07-10T09:15:39.513 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b', 'tag': None, 'wait_for_package': False} 2021-07-10T09:15:39.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo lsb_release -is 2021-07-10T09:15:39.548 INFO:teuthology.orchestra.run.smithi105.stdout:CentOSStream 2021-07-10T09:15:39.550 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-07-10T09:15:39.550 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b', 'tag': None, 'wait_for_package': False} 2021-07-10T09:15:39.551 DEBUG:teuthology.orchestra.run.smithi105:> sudo lsb_release -is 2021-07-10T09:15:39.588 INFO:teuthology.orchestra.run.smithi062.stdout:CentOSStream 2021-07-10T09:15:39.589 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-07-10T09:15:39.590 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=92adad3411b7280fb159eb9d7304c654695beb8b 2021-07-10T09:15:39.627 INFO:teuthology.orchestra.run.smithi105.stdout:CentOSStream 2021-07-10T09:15:39.628 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-07-10T09:15:39.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=92adad3411b7280fb159eb9d7304c654695beb8b 2021-07-10T09:15:39.758 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/ 2021-07-10T09:15:39.758 INFO:teuthology.task.install.rpm:Package version is 17.0.0-5861.g92adad34 2021-07-10T09:15:39.783 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/ 2021-07-10T09:15:39.784 INFO:teuthology.task.install.rpm:Package version is 17.0.0-5861.g92adad34 2021-07-10T09:15:39.989 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-07-10T09:15:39.989 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:15:39.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-07-10T09:15:39.998 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mgolub-testing/92adad3411b7280fb159eb9d7304c654695beb8b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-07-10T09:15:39.998 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:15:39.999 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-07-10T09:15:40.017 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2021-07-10T09:15:40.018 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/92adad3411b7280fb159eb9d7304c654695beb8b/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-07-10T09:15:40.024 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2021-07-10T09:15:40.024 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/92adad3411b7280fb159eb9d7304c654695beb8b/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-07-10T09:15:40.086 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-07-10T09:15:40.088 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-07-10T09:15:40.163 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-07-10T09:15:40.165 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-07-10T09:15:40.228 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2021-07-10T09:15:40.230 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2021-07-10T09:15:40.231 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2021-07-10T09:15:40.231 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2021-07-10T09:15:40.457 INFO:teuthology.orchestra.run.smithi105.stdout:49 files removed 2021-07-10T09:15:40.470 INFO:teuthology.orchestra.run.smithi062.stdout:49 files removed 2021-07-10T09:15:40.477 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2021-07-10T09:15:40.491 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw 2021-07-10T09:15:41.453 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 119 kB/s | 76 kB 00:00 2021-07-10T09:15:41.474 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 122 kB/s | 76 kB 00:00 2021-07-10T09:15:41.698 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 72 kB/s | 15 kB 00:00 2021-07-10T09:15:41.720 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 73 kB/s | 15 kB 00:00 2021-07-10T09:15:41.933 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 7.0 kB/s | 1.5 kB 00:00 2021-07-10T09:15:41.956 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 7.0 kB/s | 1.5 kB 00:00 2021-07-10T09:15:42.187 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-8 - AppStream 49 MB/s | 11 MB 00:00 2021-07-10T09:15:42.245 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-8 - AppStream 43 MB/s | 11 MB 00:00 2021-07-10T09:15:44.315 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-8 - Base 50 MB/s | 8.1 MB 00:00 2021-07-10T09:15:44.385 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-8 - Base 44 MB/s | 8.1 MB 00:00 2021-07-10T09:15:45.651 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-8 - Extras 219 kB/s | 14 kB 00:00 2021-07-10T09:15:45.702 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-8 - Extras 217 kB/s | 14 kB 00:00 2021-07-10T09:15:46.426 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 14 MB/s | 10 MB 00:00 2021-07-10T09:15:46.517 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 14 MB/s | 10 MB 00:00 2021-07-10T09:15:48.632 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 4.9 MB/s | 24 kB 00:00 2021-07-10T09:15:48.730 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-07-10T09:15:50.340 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:15:50.343 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:15:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:15:50.343 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:15:50.344 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:15:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:17.0.0-5861.g92adad34.el8 ceph 8.3 M 2021-07-10T09:15:50.344 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2021-07-10T09:15:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.3 M 2021-07-10T09:15:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.7 M 2021-07-10T09:15:50.345 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:15:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:17.0.0-5861.g92adad34.el8 ceph 5.1 M 2021-07-10T09:15:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:17.0.0-5861.g92adad34.el8 ceph 19 M 2021-07-10T09:15:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:17.0.0-5861.g92adad34.el8 ceph 24 k 2021-07-10T09:15:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-07-10T09:15:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-07-10T09:15:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-07-10T09:15:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 750 k 2021-07-10T09:15:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-07-10T09:15:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-07-10T09:15:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 452 k 2021-07-10T09:15:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-07-10T09:15:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.9 M 2021-07-10T09:15:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-07-10T09:15:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-07-10T09:15:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse 2021-07-10T09:15:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: x86_64 2:17.0.0-5861.g92adad34.el8 ceph 45 k 2021-07-10T09:15:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:17.0.0-5861.g92adad34.el8 ceph 78 k 2021-07-10T09:15:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:17.0.0-5861.g92adad34.el8 ceph 178 k 2021-07-10T09:15:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-07-10T09:15:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:17.0.0-5861.g92adad34.el8 ceph 365 k 2021-07-10T09:15:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:17.0.0-5861.g92adad34.el8 ceph 345 k 2021-07-10T09:15:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:17.0.0-5861.g92adad34.el8 ceph 109 k 2021-07-10T09:15:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:15:50.350 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:15:50.350 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:15:50.351 INFO:teuthology.orchestra.run.smithi105.stdout:Install 22 Packages 2021-07-10T09:15:50.351 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2021-07-10T09:15:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:15:50.352 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 47 M 2021-07-10T09:15:50.352 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:15:50.511 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:15:50.514 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:15:50.514 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:15:50.515 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:15:50.515 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:15:50.515 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:17.0.0-5861.g92adad34.el8 ceph 8.3 M 2021-07-10T09:15:50.515 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2021-07-10T09:15:50.516 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.3 M 2021-07-10T09:15:50.516 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.7 M 2021-07-10T09:15:50.516 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:15:50.516 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:17.0.0-5861.g92adad34.el8 ceph 5.1 M 2021-07-10T09:15:50.516 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:17.0.0-5861.g92adad34.el8 ceph 19 M 2021-07-10T09:15:50.517 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:17.0.0-5861.g92adad34.el8 ceph 24 k 2021-07-10T09:15:50.517 INFO:teuthology.orchestra.run.smithi062.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-07-10T09:15:50.517 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-07-10T09:15:50.517 INFO:teuthology.orchestra.run.smithi062.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-07-10T09:15:50.518 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 750 k 2021-07-10T09:15:50.518 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-07-10T09:15:50.518 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-07-10T09:15:50.518 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 452 k 2021-07-10T09:15:50.519 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-07-10T09:15:50.519 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.9 M 2021-07-10T09:15:50.519 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-07-10T09:15:50.519 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-07-10T09:15:50.519 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse 2021-07-10T09:15:50.520 INFO:teuthology.orchestra.run.smithi062.stdout: x86_64 2:17.0.0-5861.g92adad34.el8 ceph 45 k 2021-07-10T09:15:50.520 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:17.0.0-5861.g92adad34.el8 ceph 78 k 2021-07-10T09:15:50.520 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:17.0.0-5861.g92adad34.el8 ceph 178 k 2021-07-10T09:15:50.520 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-07-10T09:15:50.521 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:17.0.0-5861.g92adad34.el8 ceph 365 k 2021-07-10T09:15:50.521 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:17.0.0-5861.g92adad34.el8 ceph 345 k 2021-07-10T09:15:50.521 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:17.0.0-5861.g92adad34.el8 ceph 109 k 2021-07-10T09:15:50.521 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:15:50.522 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:15:50.522 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:15:50.522 INFO:teuthology.orchestra.run.smithi062.stdout:Install 22 Packages 2021-07-10T09:15:50.522 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 2 Packages 2021-07-10T09:15:50.522 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:15:50.523 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 47 M 2021-07-10T09:15:50.524 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:15:51.250 INFO:teuthology.orchestra.run.smithi105.stdout:(1/24): ceph-base-17.0.0-5861.g92adad34.el8.x86 5.7 MB/s | 5.1 MB 00:00 2021-07-10T09:15:51.296 INFO:teuthology.orchestra.run.smithi105.stdout:(2/24): ceph-selinux-17.0.0-5861.g92adad34.el8. 514 kB/s | 24 kB 00:00 2021-07-10T09:15:51.422 INFO:teuthology.orchestra.run.smithi105.stdout:(3/24): libcephfs2-17.0.0-5861.g92adad34.el8.x8 5.9 MB/s | 750 kB 00:00 2021-07-10T09:15:51.490 INFO:teuthology.orchestra.run.smithi105.stdout:(4/24): libradosstriper1-17.0.0-5861.g92adad34. 6.3 MB/s | 452 kB 00:00 2021-07-10T09:15:51.608 INFO:teuthology.orchestra.run.smithi105.stdout:(5/24): ceph-radosgw-17.0.0-5861.g92adad34.el8. 6.5 MB/s | 8.3 MB 00:01 2021-07-10T09:15:51.669 INFO:teuthology.orchestra.run.smithi105.stdout:(6/24): python3-ceph-argparse-17.0.0-5861.g92ad 737 kB/s | 45 kB 00:00 2021-07-10T09:15:51.731 INFO:teuthology.orchestra.run.smithi105.stdout:(7/24): python3-ceph-common-17.0.0-5861.g92adad 1.2 MB/s | 78 kB 00:00 2021-07-10T09:15:51.800 INFO:teuthology.orchestra.run.smithi105.stdout:(8/24): python3-cephfs-17.0.0-5861.g92adad34.el 2.5 MB/s | 178 kB 00:00 2021-07-10T09:15:51.879 INFO:teuthology.orchestra.run.smithi105.stdout:(9/24): python3-rados-17.0.0-5861.g92adad34.el8 4.5 MB/s | 365 kB 00:00 2021-07-10T09:15:51.957 INFO:teuthology.orchestra.run.smithi105.stdout:(10/24): librgw2-17.0.0-5861.g92adad34.el8.x86_ 8.3 MB/s | 3.9 MB 00:00 2021-07-10T09:15:51.959 INFO:teuthology.orchestra.run.smithi105.stdout:(11/24): python3-rbd-17.0.0-5861.g92adad34.el8. 4.2 MB/s | 345 kB 00:00 2021-07-10T09:15:51.992 INFO:teuthology.orchestra.run.smithi105.stdout:(12/24): librdkafka-0.11.4-1.el8.x86_64.rpm 11 MB/s | 353 kB 00:00 2021-07-10T09:15:51.996 INFO:teuthology.orchestra.run.smithi105.stdout:(13/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 69 MB/s | 259 kB 00:00 2021-07-10T09:15:51.997 INFO:teuthology.orchestra.run.smithi105.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 36 MB/s | 44 kB 00:00 2021-07-10T09:15:51.999 INFO:teuthology.orchestra.run.smithi105.stdout:(15/24): librabbitmq-0.9.0-3.el8.x86_64.rpm 36 MB/s | 47 kB 00:00 2021-07-10T09:15:52.007 INFO:teuthology.orchestra.run.smithi105.stdout:(16/24): python3-rgw-17.0.0-5861.g92adad34.el8. 2.1 MB/s | 109 kB 00:00 2021-07-10T09:15:52.133 INFO:teuthology.orchestra.run.smithi105.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 873 kB/s | 116 kB 00:00 2021-07-10T09:15:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:(18/24): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2021-07-10T09:15:52.187 INFO:teuthology.orchestra.run.smithi105.stdout:(19/24): leveldb-1.22-1.el8.x86_64.rpm 3.3 MB/s | 181 kB 00:00 2021-07-10T09:15:52.190 INFO:teuthology.orchestra.run.smithi105.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 2.2 MB/s | 59 kB 00:00 2021-07-10T09:15:52.215 INFO:teuthology.orchestra.run.smithi105.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2021-07-10T09:15:52.421 INFO:teuthology.orchestra.run.smithi105.stdout:(22/24): ceph-common-17.0.0-5861.g92adad34.el8. 9.2 MB/s | 19 MB 00:02 2021-07-10T09:15:52.503 INFO:teuthology.orchestra.run.smithi105.stdout:(23/24): librados2-17.0.0-5861.g92adad34.el8.x8 11 MB/s | 3.3 MB 00:00 2021-07-10T09:15:52.762 INFO:teuthology.orchestra.run.smithi105.stdout:(24/24): librbd1-17.0.0-5861.g92adad34.el8.x86_ 6.8 MB/s | 3.7 MB 00:00 2021-07-10T09:15:52.765 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:15:52.765 INFO:teuthology.orchestra.run.smithi105.stdout:Total 19 MB/s | 47 MB 00:02 2021-07-10T09:15:52.766 INFO:teuthology.orchestra.run.smithi062.stdout:(1/24): ceph-base-17.0.0-5861.g92adad34.el8.x86 2.3 MB/s | 5.1 MB 00:02 2021-07-10T09:15:52.768 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:15:52.793 INFO:teuthology.orchestra.run.smithi062.stdout:(2/24): ceph-selinux-17.0.0-5861.g92adad34.el8. 891 kB/s | 24 kB 00:00 2021-07-10T09:15:52.812 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:15:52.813 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:15:52.876 INFO:teuthology.orchestra.run.smithi062.stdout:(3/24): libcephfs2-17.0.0-5861.g92adad34.el8.x8 8.8 MB/s | 750 kB 00:00 2021-07-10T09:15:52.932 INFO:teuthology.orchestra.run.smithi062.stdout:(4/24): libradosstriper1-17.0.0-5861.g92adad34. 8.1 MB/s | 452 kB 00:00 2021-07-10T09:15:52.991 INFO:teuthology.orchestra.run.smithi062.stdout:(5/24): ceph-radosgw-17.0.0-5861.g92adad34.el8. 3.3 MB/s | 8.3 MB 00:02 2021-07-10T09:15:53.020 INFO:teuthology.orchestra.run.smithi062.stdout:(6/24): python3-ceph-argparse-17.0.0-5861.g92ad 1.5 MB/s | 45 kB 00:00 2021-07-10T09:15:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:(7/24): python3-ceph-common-17.0.0-5861.g92adad 3.0 MB/s | 78 kB 00:00 2021-07-10T09:15:53.074 INFO:teuthology.orchestra.run.smithi062.stdout:(8/24): python3-cephfs-17.0.0-5861.g92adad34.el 6.1 MB/s | 178 kB 00:00 2021-07-10T09:15:53.109 INFO:teuthology.orchestra.run.smithi062.stdout:(9/24): python3-rados-17.0.0-5861.g92adad34.el8 10 MB/s | 365 kB 00:00 2021-07-10T09:15:53.147 INFO:teuthology.orchestra.run.smithi062.stdout:(10/24): python3-rbd-17.0.0-5861.g92adad34.el8. 9.3 MB/s | 345 kB 00:00 2021-07-10T09:15:53.158 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:15:53.161 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:15:53.174 INFO:teuthology.orchestra.run.smithi062.stdout:(11/24): python3-rgw-17.0.0-5861.g92adad34.el8. 3.9 MB/s | 109 kB 00:00 2021-07-10T09:15:53.194 INFO:teuthology.orchestra.run.smithi062.stdout:(12/24): librgw2-17.0.0-5861.g92adad34.el8.x86_ 15 MB/s | 3.9 MB 00:00 2021-07-10T09:15:53.212 INFO:teuthology.orchestra.run.smithi062.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 9.4 MB/s | 353 kB 00:00 2021-07-10T09:15:53.212 INFO:teuthology.orchestra.run.smithi062.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 35 MB/s | 44 kB 00:00 2021-07-10T09:15:53.214 INFO:teuthology.orchestra.run.smithi062.stdout:(15/24): librabbitmq-0.9.0-3.el8.x86_64.rpm 32 MB/s | 47 kB 00:00 2021-07-10T09:15:53.225 INFO:teuthology.orchestra.run.smithi062.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.1 MB/s | 259 kB 00:00 2021-07-10T09:15:53.349 INFO:teuthology.orchestra.run.smithi062.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 866 kB/s | 116 kB 00:00 2021-07-10T09:15:53.379 INFO:teuthology.orchestra.run.smithi062.stdout:(18/24): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2021-07-10T09:15:53.403 INFO:teuthology.orchestra.run.smithi062.stdout:(19/24): liboath-2.6.2-3.el8.x86_64.rpm 2.5 MB/s | 59 kB 00:00 2021-07-10T09:15:53.404 INFO:teuthology.orchestra.run.smithi062.stdout:(20/24): leveldb-1.22-1.el8.x86_64.rpm 3.2 MB/s | 181 kB 00:00 2021-07-10T09:15:53.426 INFO:teuthology.orchestra.run.smithi062.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 3.2 MB/s | 75 kB 00:00 2021-07-10T09:15:53.649 INFO:teuthology.orchestra.run.smithi062.stdout:(22/24): librados2-17.0.0-5861.g92adad34.el8.x8 14 MB/s | 3.3 MB 00:00 2021-07-10T09:15:53.707 INFO:teuthology.orchestra.run.smithi062.stdout:(23/24): librbd1-17.0.0-5861.g92adad34.el8.x86_ 13 MB/s | 3.7 MB 00:00 2021-07-10T09:15:54.137 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:15:54.296 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-07-10T09:15:54.487 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-07-10T09:15:54.598 INFO:teuthology.orchestra.run.smithi062.stdout:(24/24): ceph-common-17.0.0-5861.g92adad34.el8. 4.7 MB/s | 19 MB 00:04 2021-07-10T09:15:54.603 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:15:54.604 INFO:teuthology.orchestra.run.smithi062.stdout:Total 12 MB/s | 47 MB 00:04 2021-07-10T09:15:54.609 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:15:54.632 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-07-10T09:15:54.663 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:15:54.663 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:15:55.024 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:15:55.027 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:15:55.034 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-07-10T09:15:55.190 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:17.0.0-5861.g92adad34.el8.x86_64 4/26 2021-07-10T09:15:55.283 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:17.0.0-5861.g92adad34.el8.x86_64 4/26 2021-07-10T09:15:55.486 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 5/26 2021-07-10T09:15:55.690 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 6/26 2021-07-10T09:15:56.100 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:15:56.102 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 6/26 2021-07-10T09:15:56.256 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-07-10T09:15:56.265 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 7/26 2021-07-10T09:15:56.331 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 7/26 2021-07-10T09:15:56.452 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-07-10T09:15:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/26 2021-07-10T09:15:56.604 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-07-10T09:15:56.681 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-07-10T09:15:57.000 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-07-10T09:15:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-07-10T09:15:57.195 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:17.0.0-5861.g92adad34.el8.x86_64 4/26 2021-07-10T09:15:57.223 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 10/26 2021-07-10T09:15:57.299 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:17.0.0-5861.g92adad34.el8.x86_64 4/26 2021-07-10T09:15:57.304 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 10/26 2021-07-10T09:15:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-07-10T09:15:57.574 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:17.0.0-5861.g92adad34.el8. 12/26 2021-07-10T09:15:57.653 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 5/26 2021-07-10T09:15:57.727 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 13/26 2021-07-10T09:15:57.820 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 6/26 2021-07-10T09:15:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 14/26 2021-07-10T09:15:58.097 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 15/26 2021-07-10T09:15:58.250 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 6/26 2021-07-10T09:15:58.264 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_6 16/26 2021-07-10T09:15:58.341 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_6 16/26 2021-07-10T09:15:58.545 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 7/26 2021-07-10T09:15:58.611 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 7/26 2021-07-10T09:15:58.800 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-07-10T09:15:58.801 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/26 2021-07-10T09:15:58.949 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2021-07-10T09:15:58.970 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-07-10T09:15:59.130 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-07-10T09:15:59.290 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:17.0.0-5861.g92adad34.el8.x8 20/26 2021-07-10T09:15:59.388 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-07-10T09:15:59.562 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 10/26 2021-07-10T09:15:59.626 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 10/26 2021-07-10T09:15:59.802 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-07-10T09:15:59.960 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:17.0.0-5861.g92adad34.el8. 12/26 2021-07-10T09:16:00.134 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 13/26 2021-07-10T09:16:00.307 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 14/26 2021-07-10T09:16:00.505 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 15/26 2021-07-10T09:16:00.695 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_6 16/26 2021-07-10T09:16:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_6 16/26 2021-07-10T09:16:00.975 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-07-10T09:16:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2021-07-10T09:16:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-07-10T09:16:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:17.0.0-5861.g92adad34.el8.x8 20/26 2021-07-10T09:16:03.219 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 21/26 2021-07-10T09:16:03.431 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 21/26 2021-07-10T09:16:04.024 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 21/26 2021-07-10T09:16:04.320 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 22/26 2021-07-10T09:16:04.412 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 22/26 2021-07-10T09:16:04.522 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 23/26 2021-07-10T09:16:05.668 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 21/26 2021-07-10T09:16:05.887 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 21/26 2021-07-10T09:16:06.494 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 21/26 2021-07-10T09:16:06.748 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 22/26 2021-07-10T09:16:06.841 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 22/26 2021-07-10T09:16:06.936 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 23/26 2021-07-10T09:16:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 23/26 2021-07-10T09:16:22.703 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2021-07-10T09:16:22.704 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2021-07-10T09:16:22.704 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2021-07-10T09:16:22.704 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /run 2021-07-10T09:16:22.704 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2021-07-10T09:16:22.705 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2021-07-10T09:16:22.705 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2021-07-10T09:16:22.705 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2021-07-10T09:16:22.705 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2021-07-10T09:16:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:23.776 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 24/26 2021-07-10T09:16:23.853 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 24/26 2021-07-10T09:16:23.854 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-07-10T09:16:24.063 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-07-10T09:16:24.063 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-07-10T09:16:24.810 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-07-10T09:16:24.811 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 1/26 2021-07-10T09:16:24.811 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 2/26 2021-07-10T09:16:24.811 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 3/26 2021-07-10T09:16:24.812 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 4/26 2021-07-10T09:16:24.812 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 5/26 2021-07-10T09:16:24.812 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_6 6/26 2021-07-10T09:16:24.813 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 7/26 2021-07-10T09:16:24.813 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:17.0.0-5861.g92adad34.el8. 8/26 2021-07-10T09:16:24.813 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:17.0.0-5861.g92adad34.el8.x8 9/26 2021-07-10T09:16:24.813 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 10/26 2021-07-10T09:16:24.814 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 11/26 2021-07-10T09:16:24.814 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 12/26 2021-07-10T09:16:24.815 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 13/26 2021-07-10T09:16:24.816 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-07-10T09:16:24.816 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-07-10T09:16:24.817 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-07-10T09:16:24.817 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/26 2021-07-10T09:16:24.818 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-07-10T09:16:24.819 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2021-07-10T09:16:24.819 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-07-10T09:16:24.819 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-07-10T09:16:24.820 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-07-10T09:16:24.820 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:17.0.0-5861.g92adad34.el8.x86_64 23/26 2021-07-10T09:16:24.821 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-07-10T09:16:24.821 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 25/26 2021-07-10T09:16:25.724 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-07-10T09:16:25.724 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:25.725 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2021-07-10T09:16:25.725 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.725 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.725 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:16:25.726 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.726 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.726 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.726 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.727 INFO:teuthology.orchestra.run.smithi105.stdout: fmt-6.2.1-1.el8.x86_64 2021-07-10T09:16:25.727 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-07-10T09:16:25.727 INFO:teuthology.orchestra.run.smithi105.stdout: leveldb-1.22-1.el8.x86_64 2021-07-10T09:16:25.728 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.728 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.2-3.el8.x86_64 2021-07-10T09:16:25.728 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-07-10T09:16:25.728 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.729 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-07-10T09:16:25.729 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.729 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.3.1-3.el8.x86_64 2021-07-10T09:16:25.729 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-07-10T09:16:25.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-07-10T09:16:25.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:25.732 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:25.732 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:16:25.835 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 23/26 2021-07-10T09:16:25.836 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2021-07-10T09:16:25.836 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2021-07-10T09:16:25.836 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /dev 2021-07-10T09:16:25.837 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /run 2021-07-10T09:16:25.837 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2021-07-10T09:16:25.837 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2021-07-10T09:16:25.837 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2021-07-10T09:16:25.838 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2021-07-10T09:16:25.838 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /dev 2021-07-10T09:16:25.838 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:25.966 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2021-07-10T09:16:26.444 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:38 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:27.003 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 24/26 2021-07-10T09:16:27.070 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 24/26 2021-07-10T09:16:27.070 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-07-10T09:16:27.201 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:16:27.202 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:27.202 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:16:27.202 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:27.202 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:16:27.203 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:17.0.0-5861.g92adad34.el8 ceph 44 M 2021-07-10T09:16:27.203 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:16:27.203 INFO:teuthology.orchestra.run.smithi105.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-07-10T09:16:27.203 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:17.0.0-5861.g92adad34.el8 ceph 157 k 2021-07-10T09:16:27.204 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-07-10T09:16:27.204 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-07-10T09:16:27.204 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-07-10T09:16:27.204 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:27.205 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:16:27.205 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:27.205 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2021-07-10T09:16:27.205 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:27.206 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 45 M 2021-07-10T09:16:27.206 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 207 M 2021-07-10T09:16:27.207 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:16:27.243 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.0 MB/s | 161 kB 00:00 2021-07-10T09:16:27.247 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 51 MB/s | 187 kB 00:00 2021-07-10T09:16:27.252 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 60 MB/s | 323 kB 00:00 2021-07-10T09:16:27.261 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-07-10T09:16:27.261 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-07-10T09:16:27.342 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 773 kB/s | 69 kB 00:00 2021-07-10T09:16:27.377 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): libcephsqlite-17.0.0-5861.g92adad34.el8. 909 kB/s | 157 kB 00:00 2021-07-10T09:16:28.024 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-07-10T09:16:28.025 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 1/26 2021-07-10T09:16:28.025 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 2/26 2021-07-10T09:16:28.025 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 3/26 2021-07-10T09:16:28.025 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 4/26 2021-07-10T09:16:28.026 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 5/26 2021-07-10T09:16:28.026 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_6 6/26 2021-07-10T09:16:28.026 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 7/26 2021-07-10T09:16:28.026 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:17.0.0-5861.g92adad34.el8. 8/26 2021-07-10T09:16:28.026 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:17.0.0-5861.g92adad34.el8.x8 9/26 2021-07-10T09:16:28.027 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 10/26 2021-07-10T09:16:28.027 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 11/26 2021-07-10T09:16:28.027 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 12/26 2021-07-10T09:16:28.027 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 13/26 2021-07-10T09:16:28.028 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-07-10T09:16:28.028 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-07-10T09:16:28.028 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-07-10T09:16:28.028 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/26 2021-07-10T09:16:28.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-07-10T09:16:28.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2021-07-10T09:16:28.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-07-10T09:16:28.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-07-10T09:16:28.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-07-10T09:16:28.030 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:17.0.0-5861.g92adad34.el8.x86_64 23/26 2021-07-10T09:16:28.030 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-07-10T09:16:28.030 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 25/26 2021-07-10T09:16:28.895 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-07-10T09:16:28.895 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:28.896 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2021-07-10T09:16:28.896 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.896 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.896 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:16:28.897 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.897 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.897 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.897 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.897 INFO:teuthology.orchestra.run.smithi062.stdout: fmt-6.2.1-1.el8.x86_64 2021-07-10T09:16:28.898 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-07-10T09:16:28.898 INFO:teuthology.orchestra.run.smithi062.stdout: leveldb-1.22-1.el8.x86_64 2021-07-10T09:16:28.898 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.899 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.2-3.el8.x86_64 2021-07-10T09:16:28.899 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-07-10T09:16:28.899 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.899 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-07-10T09:16:28.899 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.900 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.3.1-3.el8.x86_64 2021-07-10T09:16:28.900 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-07-10T09:16:28.900 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.900 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.901 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.901 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-07-10T09:16:28.901 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.901 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.902 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:28.902 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:28.902 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:16:29.140 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-test 2021-07-10T09:16:29.591 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): ceph-test-17.0.0-5861.g92adad34.el8.x86_ 18 MB/s | 44 MB 00:02 2021-07-10T09:16:29.595 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:16:29.596 INFO:teuthology.orchestra.run.smithi105.stdout:Total 19 MB/s | 45 MB 00:02 2021-07-10T09:16:29.599 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:16:29.621 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:16:29.622 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:16:29.644 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:41 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:29.870 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:16:29.873 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:16:30.352 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:16:30.560 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-07-10T09:16:30.613 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:16:30.614 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:16:30.614 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:16:30.614 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:16:30.614 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:16:30.615 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:17.0.0-5861.g92adad34.el8 ceph 44 M 2021-07-10T09:16:30.615 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:16:30.615 INFO:teuthology.orchestra.run.smithi062.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-07-10T09:16:30.615 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:17.0.0-5861.g92adad34.el8 ceph 157 k 2021-07-10T09:16:30.615 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-07-10T09:16:30.616 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-07-10T09:16:30.616 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-07-10T09:16:30.616 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:30.616 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:16:30.617 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:16:30.617 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2021-07-10T09:16:30.617 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:30.618 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 45 M 2021-07-10T09:16:30.618 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 207 M 2021-07-10T09:16:30.618 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:16:30.649 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.8 MB/s | 161 kB 00:00 2021-07-10T09:16:30.655 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 48 MB/s | 187 kB 00:00 2021-07-10T09:16:30.659 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 62 MB/s | 323 kB 00:00 2021-07-10T09:16:30.748 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 773 kB/s | 69 kB 00:00 2021-07-10T09:16:30.759 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-07-10T09:16:30.935 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-07-10T09:16:31.015 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-07-10T09:16:31.109 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): libcephsqlite-17.0.0-5861.g92adad34.el8. 319 kB/s | 157 kB 00:00 2021-07-10T09:16:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : jq-1.5-12.el8.x86_64 4/6 2021-07-10T09:16:31.302 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 5/6 2021-07-10T09:16:33.881 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): ceph-test-17.0.0-5861.g92adad34.el8.x86_ 13 MB/s | 44 MB 00:03 2021-07-10T09:16:33.885 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:16:33.885 INFO:teuthology.orchestra.run.smithi062.stdout:Total 14 MB/s | 45 MB 00:03 2021-07-10T09:16:33.889 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:16:33.915 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:16:33.915 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:16:34.179 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:16:34.182 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:16:34.643 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:16:34.866 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-07-10T09:16:35.087 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-07-10T09:16:35.250 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-07-10T09:16:35.359 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-07-10T09:16:35.514 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : jq-1.5-12.el8.x86_64 4/6 2021-07-10T09:16:35.610 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 5/6 2021-07-10T09:16:35.640 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 5/6 2021-07-10T09:16:35.833 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 6/6 2021-07-10T09:16:36.082 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 6/6 2021-07-10T09:16:36.082 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 1/6 2021-07-10T09:16:36.082 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 2/6 2021-07-10T09:16:36.082 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : jq-1.5-12.el8.x86_64 3/6 2021-07-10T09:16:36.083 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-07-10T09:16:36.083 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-07-10T09:16:36.529 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-07-10T09:16:36.529 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:36.530 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:16:36.530 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:36.530 INFO:teuthology.orchestra.run.smithi105.stdout: jq-1.5-12.el8.x86_64 2021-07-10T09:16:36.530 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:36.531 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-07-10T09:16:36.531 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-1.el8.x86_64 2021-07-10T09:16:36.531 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-07-10T09:16:36.532 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:36.532 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:16:36.720 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2021-07-10T09:16:37.192 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:49 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:37.954 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:16:37.958 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2021-07-10T09:16:37.958 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:16:37.958 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2021-07-10T09:16:37.958 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:16:37.959 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:17.0.0-5861.g92adad34.el8 ceph 6.4 k 2021-07-10T09:16:37.959 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:16:37.959 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:17.0.0-5861.g92adad34.el8 ceph 2.2 M 2021-07-10T09:16:37.959 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:17.0.0-5861.g92adad34.el8 ceph 1.5 M 2021-07-10T09:16:37.960 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 208 k 2021-07-10T09:16:37.960 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.8 M 2021-07-10T09:16:37.960 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:17.0.0-5861.g92adad34.el8 ceph 17 M 2021-07-10T09:16:37.960 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-07-10T09:16:37.961 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-07-10T09:16:37.961 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-07-10T09:16:37.961 INFO:teuthology.orchestra.run.smithi105.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-07-10T09:16:37.961 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-07-10T09:16:37.961 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-07-10T09:16:37.962 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-07-10T09:16:37.962 INFO:teuthology.orchestra.run.smithi105.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-07-10T09:16:37.962 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-07-10T09:16:37.962 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-07-10T09:16:37.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-07-10T09:16:37.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-07-10T09:16:37.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-07-10T09:16:37.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-07-10T09:16:37.964 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-07-10T09:16:37.964 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-07-10T09:16:37.964 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-07-10T09:16:37.964 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-07-10T09:16:37.964 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-07-10T09:16:37.965 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-07-10T09:16:37.965 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-07-10T09:16:37.965 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-07-10T09:16:37.965 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-07-10T09:16:37.966 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-07-10T09:16:37.966 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-07-10T09:16:37.966 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-07-10T09:16:37.966 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-07-10T09:16:37.966 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-07-10T09:16:37.967 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-07-10T09:16:37.967 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-07-10T09:16:37.967 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-07-10T09:16:37.967 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:37.968 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:16:37.968 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2021-07-10T09:16:37.968 INFO:teuthology.orchestra.run.smithi105.stdout:Install 37 Packages 2021-07-10T09:16:37.968 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:37.969 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 29 M 2021-07-10T09:16:37.969 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 110 M 2021-07-10T09:16:37.970 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:16:38.068 INFO:teuthology.orchestra.run.smithi105.stdout:(1/37): ceph-17.0.0-5861.g92adad34.el8.x86_64.r 62 kB/s | 6.4 kB 00:00 2021-07-10T09:16:38.318 INFO:teuthology.orchestra.run.smithi105.stdout:(2/37): ceph-mgr-17.0.0-5861.g92adad34.el8.x86_ 4.2 MB/s | 1.5 MB 00:00 2021-07-10T09:16:38.359 INFO:teuthology.orchestra.run.smithi105.stdout:(3/37): ceph-mds-17.0.0-5861.g92adad34.el8.x86_ 5.6 MB/s | 2.2 MB 00:00 2021-07-10T09:16:38.411 INFO:teuthology.orchestra.run.smithi105.stdout:(4/37): ceph-mgr-modules-core-17.0.0-5861.g92ad 4.0 MB/s | 208 kB 00:00 2021-07-10T09:16:38.443 INFO:teuthology.orchestra.run.smithi105.stdout:(5/37): python3-mako-1.0.6-13.el8.noarch.rpm 4.7 MB/s | 157 kB 00:00 2021-07-10T09:16:38.445 INFO:teuthology.orchestra.run.smithi105.stdout:(6/37): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2021-07-10T09:16:38.447 INFO:teuthology.orchestra.run.smithi105.stdout:(7/37): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 48 MB/s | 103 kB 00:00 2021-07-10T09:16:38.449 INFO:teuthology.orchestra.run.smithi105.stdout:(8/37): python3-pytz-2017.2-9.el8.noarch.rpm 29 MB/s | 54 kB 00:00 2021-07-10T09:16:38.456 INFO:teuthology.orchestra.run.smithi105.stdout:(9/37): python3-werkzeug-0.12.2-4.el8.noarch.rp 66 MB/s | 457 kB 00:00 2021-07-10T09:16:38.460 INFO:teuthology.orchestra.run.smithi105.stdout:(10/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 67 MB/s | 237 kB 00:00 2021-07-10T09:16:38.463 INFO:teuthology.orchestra.run.smithi105.stdout:(11/37): python3-chardet-3.0.4-7.el8.noarch.rpm 65 MB/s | 195 kB 00:00 2021-07-10T09:16:38.472 INFO:teuthology.orchestra.run.smithi105.stdout:(12/37): python3-cryptography-3.2.1-5.el8.x86_6 65 MB/s | 559 kB 00:00 2021-07-10T09:16:38.474 INFO:teuthology.orchestra.run.smithi105.stdout:(13/37): python3-idna-2.5-5.el8.noarch.rpm 48 MB/s | 97 kB 00:00 2021-07-10T09:16:38.477 INFO:teuthology.orchestra.run.smithi105.stdout:(14/37): python3-pycparser-2.14-14.el8.noarch.r 48 MB/s | 109 kB 00:00 2021-07-10T09:16:38.478 INFO:teuthology.orchestra.run.smithi105.stdout:(15/37): python3-pysocks-1.6.8-3.el8.noarch.rpm 32 MB/s | 34 kB 00:00 2021-07-10T09:16:38.480 INFO:teuthology.orchestra.run.smithi105.stdout:(16/37): python3-requests-2.20.0-2.1.el8_1.noar 59 MB/s | 123 kB 00:00 2021-07-10T09:16:38.483 INFO:teuthology.orchestra.run.smithi105.stdout:(17/37): python3-urllib3-1.24.2-5.el8.noarch.rp 65 MB/s | 177 kB 00:00 2021-07-10T09:16:38.546 INFO:teuthology.orchestra.run.smithi105.stdout:(18/37): ceph-mon-17.0.0-5861.g92adad34.el8.x86 8.0 MB/s | 3.8 MB 00:00 2021-07-10T09:16:38.653 INFO:teuthology.orchestra.run.smithi105.stdout:(19/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 257 kB/s | 44 kB 00:00 2021-07-10T09:16:38.686 INFO:teuthology.orchestra.run.smithi105.stdout:(20/37): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2021-07-10T09:16:38.732 INFO:teuthology.orchestra.run.smithi105.stdout:(21/37): python3-cherrypy-18.4.0-1.el8.noarch.r 8.3 MB/s | 384 kB 00:00 2021-07-10T09:16:38.733 INFO:teuthology.orchestra.run.smithi105.stdout:(22/37): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.1 MB/s | 173 kB 00:00 2021-07-10T09:16:38.754 INFO:teuthology.orchestra.run.smithi105.stdout:(23/37): python3-jaraco-6.2-6.el8.noarch.rpm 496 kB/s | 11 kB 00:00 2021-07-10T09:16:38.760 INFO:teuthology.orchestra.run.smithi105.stdout:(24/37): python3-jaraco-functools-2.0-4.el8.noa 654 kB/s | 18 kB 00:00 2021-07-10T09:16:38.777 INFO:teuthology.orchestra.run.smithi105.stdout:(25/37): python3-logutils-0.3.5-11.el8.noarch.r 2.1 MB/s | 49 kB 00:00 2021-07-10T09:16:38.788 INFO:teuthology.orchestra.run.smithi105.stdout:(26/37): python3-more-itertools-7.2.0-3.el8.noa 2.1 MB/s | 59 kB 00:00 2021-07-10T09:16:38.802 INFO:teuthology.orchestra.run.smithi105.stdout:(27/37): python3-pecan-1.3.2-9.el8.noarch.rpm 11 MB/s | 283 kB 00:00 2021-07-10T09:16:38.815 INFO:teuthology.orchestra.run.smithi105.stdout:(28/37): python3-portend-2.6-1.el8.noarch.rpm 581 kB/s | 16 kB 00:00 2021-07-10T09:16:38.824 INFO:teuthology.orchestra.run.smithi105.stdout:(29/37): python3-simplegeneric-0.8.1-17.el8.noa 844 kB/s | 19 kB 00:00 2021-07-10T09:16:38.842 INFO:teuthology.orchestra.run.smithi105.stdout:(30/37): python3-singledispatch-3.4.0.3-18.el8. 889 kB/s | 24 kB 00:00 2021-07-10T09:16:38.847 INFO:teuthology.orchestra.run.smithi105.stdout:(31/37): python3-tempora-1.14.1-5.el8.noarch.rp 1.3 MB/s | 29 kB 00:00 2021-07-10T09:16:38.870 INFO:teuthology.orchestra.run.smithi105.stdout:(32/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2021-07-10T09:16:38.871 INFO:teuthology.orchestra.run.smithi105.stdout:(33/37): python3-waitress-1.2.1-2.el8.1.noarch. 9.6 MB/s | 241 kB 00:00 2021-07-10T09:16:38.894 INFO:teuthology.orchestra.run.smithi105.stdout:(34/37): python3-webtest-2.0.33-1.el8.noarch.rp 3.7 MB/s | 85 kB 00:00 2021-07-10T09:16:38.916 INFO:teuthology.orchestra.run.smithi105.stdout:(35/37): python3-zc-lockfile-2.0-2.el8.noarch.r 1.0 MB/s | 23 kB 00:00 2021-07-10T09:16:38.924 INFO:teuthology.orchestra.run.smithi105.stdout:(36/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.5 MB/s | 251 kB 00:00 2021-07-10T09:16:39.265 INFO:teuthology.orchestra.run.smithi105.stdout:(37/37): ceph-osd-17.0.0-5861.g92adad34.el8.x86 18 MB/s | 17 MB 00:00 2021-07-10T09:16:39.268 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:16:39.269 INFO:teuthology.orchestra.run.smithi105.stdout:Total 22 MB/s | 29 MB 00:01 2021-07-10T09:16:39.279 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:16:39.318 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:16:39.319 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:16:39.620 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:16:39.623 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:16:40.243 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 5/6 2021-07-10T09:16:40.490 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 6/6 2021-07-10T09:16:40.753 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 6/6 2021-07-10T09:16:40.754 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 1/6 2021-07-10T09:16:40.754 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 2/6 2021-07-10T09:16:40.754 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : jq-1.5-12.el8.x86_64 3/6 2021-07-10T09:16:40.754 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-07-10T09:16:40.755 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-07-10T09:16:40.856 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:16:41.075 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-07-10T09:16:41.155 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-07-10T09:16:41.156 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:41.156 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:16:41.156 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:41.156 INFO:teuthology.orchestra.run.smithi062.stdout: jq-1.5-12.el8.x86_64 2021-07-10T09:16:41.157 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:41.157 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-07-10T09:16:41.157 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-1.el8.x86_64 2021-07-10T09:16:41.157 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-07-10T09:16:41.158 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:41.158 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:16:41.270 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-07-10T09:16:41.321 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph 2021-07-10T09:16:41.425 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/37 2021-07-10T09:16:41.625 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/37 2021-07-10T09:16:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/37 2021-07-10T09:16:41.813 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:53 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:41.971 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/37 2021-07-10T09:16:42.118 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/37 2021-07-10T09:16:42.253 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/37 2021-07-10T09:16:42.389 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/37 2021-07-10T09:16:42.605 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/37 2021-07-10T09:16:42.651 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:16:42.658 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2021-07-10T09:16:42.658 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:16:42.658 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2021-07-10T09:16:42.658 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:16:42.659 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:17.0.0-5861.g92adad34.el8 ceph 6.4 k 2021-07-10T09:16:42.659 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:16:42.659 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:17.0.0-5861.g92adad34.el8 ceph 2.2 M 2021-07-10T09:16:42.659 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:17.0.0-5861.g92adad34.el8 ceph 1.5 M 2021-07-10T09:16:42.660 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 208 k 2021-07-10T09:16:42.660 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.8 M 2021-07-10T09:16:42.660 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:17.0.0-5861.g92adad34.el8 ceph 17 M 2021-07-10T09:16:42.660 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-07-10T09:16:42.661 INFO:teuthology.orchestra.run.smithi062.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-07-10T09:16:42.662 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-07-10T09:16:42.662 INFO:teuthology.orchestra.run.smithi062.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-07-10T09:16:42.662 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-07-10T09:16:42.663 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-07-10T09:16:42.663 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-07-10T09:16:42.663 INFO:teuthology.orchestra.run.smithi062.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-07-10T09:16:42.663 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-07-10T09:16:42.663 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-07-10T09:16:42.664 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-07-10T09:16:42.664 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-07-10T09:16:42.664 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-07-10T09:16:42.665 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-07-10T09:16:42.665 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-07-10T09:16:42.665 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-07-10T09:16:42.666 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-07-10T09:16:42.666 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-07-10T09:16:42.666 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-07-10T09:16:42.666 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-07-10T09:16:42.667 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-07-10T09:16:42.667 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-07-10T09:16:42.667 INFO:teuthology.orchestra.run.smithi062.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-07-10T09:16:42.667 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-07-10T09:16:42.667 INFO:teuthology.orchestra.run.smithi062.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-07-10T09:16:42.668 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-07-10T09:16:42.668 INFO:teuthology.orchestra.run.smithi062.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-07-10T09:16:42.668 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-07-10T09:16:42.668 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-07-10T09:16:42.669 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-07-10T09:16:42.669 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-07-10T09:16:42.669 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:42.669 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:16:42.669 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2021-07-10T09:16:42.670 INFO:teuthology.orchestra.run.smithi062.stdout:Install 37 Packages 2021-07-10T09:16:42.670 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:42.670 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 29 M 2021-07-10T09:16:42.670 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 110 M 2021-07-10T09:16:42.671 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:16:42.767 INFO:teuthology.orchestra.run.smithi062.stdout:(1/37): ceph-17.0.0-5861.g92adad34.el8.x86_64.r 61 kB/s | 6.4 kB 00:00 2021-07-10T09:16:42.809 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 11/37 2021-07-10T09:16:42.971 INFO:teuthology.orchestra.run.smithi062.stdout:(2/37): ceph-mgr-17.0.0-5861.g92adad34.el8.x86_ 4.8 MB/s | 1.5 MB 00:00 2021-07-10T09:16:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 12/37 2021-07-10T09:16:43.059 INFO:teuthology.orchestra.run.smithi062.stdout:(3/37): ceph-mds-17.0.0-5861.g92adad34.el8.x86_ 5.5 MB/s | 2.2 MB 00:00 2021-07-10T09:16:43.109 INFO:teuthology.orchestra.run.smithi062.stdout:(4/37): ceph-mgr-modules-core-17.0.0-5861.g92ad 4.0 MB/s | 208 kB 00:00 2021-07-10T09:16:43.140 INFO:teuthology.orchestra.run.smithi062.stdout:(5/37): python3-mako-1.0.6-13.el8.noarch.rpm 5.0 MB/s | 157 kB 00:00 2021-07-10T09:16:43.141 INFO:teuthology.orchestra.run.smithi062.stdout:(6/37): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2021-07-10T09:16:43.143 INFO:teuthology.orchestra.run.smithi062.stdout:(7/37): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 53 MB/s | 103 kB 00:00 2021-07-10T09:16:43.145 INFO:teuthology.orchestra.run.smithi062.stdout:(8/37): python3-pytz-2017.2-9.el8.noarch.rpm 38 MB/s | 54 kB 00:00 2021-07-10T09:16:43.151 INFO:teuthology.orchestra.run.smithi062.stdout:(9/37): python3-werkzeug-0.12.2-4.el8.noarch.rp 74 MB/s | 457 kB 00:00 2021-07-10T09:16:43.155 INFO:teuthology.orchestra.run.smithi062.stdout:(10/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 66 MB/s | 237 kB 00:00 2021-07-10T09:16:43.159 INFO:teuthology.orchestra.run.smithi062.stdout:(11/37): python3-chardet-3.0.4-7.el8.noarch.rpm 64 MB/s | 195 kB 00:00 2021-07-10T09:16:43.166 INFO:teuthology.orchestra.run.smithi062.stdout:(12/37): python3-cryptography-3.2.1-5.el8.x86_6 76 MB/s | 559 kB 00:00 2021-07-10T09:16:43.168 INFO:teuthology.orchestra.run.smithi062.stdout:(13/37): python3-idna-2.5-5.el8.noarch.rpm 50 MB/s | 97 kB 00:00 2021-07-10T09:16:43.170 INFO:teuthology.orchestra.run.smithi062.stdout:(14/37): python3-pycparser-2.14-14.el8.noarch.r 53 MB/s | 109 kB 00:00 2021-07-10T09:16:43.171 INFO:teuthology.orchestra.run.smithi062.stdout:(15/37): python3-pysocks-1.6.8-3.el8.noarch.rpm 31 MB/s | 34 kB 00:00 2021-07-10T09:16:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:(16/37): python3-requests-2.20.0-2.1.el8_1.noar 59 MB/s | 123 kB 00:00 2021-07-10T09:16:43.177 INFO:teuthology.orchestra.run.smithi062.stdout:(17/37): python3-urllib3-1.24.2-5.el8.noarch.rp 64 MB/s | 177 kB 00:00 2021-07-10T09:16:43.203 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 13/37 2021-07-10T09:16:43.310 INFO:teuthology.orchestra.run.smithi062.stdout:(18/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 328 kB/s | 44 kB 00:00 2021-07-10T09:16:43.377 INFO:teuthology.orchestra.run.smithi062.stdout:(19/37): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.7 MB/s | 185 kB 00:00 2021-07-10T09:16:43.401 INFO:teuthology.orchestra.run.smithi062.stdout:(20/37): python3-cheroot-8.5.2-1.el8.noarch.rpm 7.2 MB/s | 173 kB 00:00 2021-07-10T09:16:43.423 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 14/37 2021-07-10T09:16:43.446 INFO:teuthology.orchestra.run.smithi062.stdout:(21/37): python3-cherrypy-18.4.0-1.el8.noarch.r 8.3 MB/s | 384 kB 00:00 2021-07-10T09:16:43.462 INFO:teuthology.orchestra.run.smithi062.stdout:(22/37): ceph-mon-17.0.0-5861.g92adad34.el8.x86 5.5 MB/s | 3.8 MB 00:00 2021-07-10T09:16:43.468 INFO:teuthology.orchestra.run.smithi062.stdout:(23/37): python3-jaraco-6.2-6.el8.noarch.rpm 501 kB/s | 11 kB 00:00 2021-07-10T09:16:43.491 INFO:teuthology.orchestra.run.smithi062.stdout:(24/37): python3-logutils-0.3.5-11.el8.noarch.r 2.1 MB/s | 49 kB 00:00 2021-07-10T09:16:43.514 INFO:teuthology.orchestra.run.smithi062.stdout:(25/37): python3-more-itertools-7.2.0-3.el8.noa 2.6 MB/s | 59 kB 00:00 2021-07-10T09:16:43.539 INFO:teuthology.orchestra.run.smithi062.stdout:(26/37): python3-pecan-1.3.2-9.el8.noarch.rpm 11 MB/s | 283 kB 00:00 2021-07-10T09:16:43.552 INFO:teuthology.orchestra.run.smithi062.stdout:(27/37): python3-jaraco-functools-2.0-4.el8.noa 195 kB/s | 18 kB 00:00 2021-07-10T09:16:43.561 INFO:teuthology.orchestra.run.smithi062.stdout:(28/37): python3-portend-2.6-1.el8.noarch.rpm 705 kB/s | 16 kB 00:00 2021-07-10T09:16:43.576 INFO:teuthology.orchestra.run.smithi062.stdout:(29/37): python3-simplegeneric-0.8.1-17.el8.noa 837 kB/s | 19 kB 00:00 2021-07-10T09:16:43.583 INFO:teuthology.orchestra.run.smithi062.stdout:(30/37): python3-singledispatch-3.4.0.3-18.el8. 1.0 MB/s | 24 kB 00:00 2021-07-10T09:16:43.598 INFO:teuthology.orchestra.run.smithi062.stdout:(31/37): python3-tempora-1.14.1-5.el8.noarch.rp 1.3 MB/s | 29 kB 00:00 2021-07-10T09:16:43.607 INFO:teuthology.orchestra.run.smithi062.stdout:(32/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.2 MB/s | 27 kB 00:00 2021-07-10T09:16:43.631 INFO:teuthology.orchestra.run.smithi062.stdout:(33/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 10 MB/s | 251 kB 00:00 2021-07-10T09:16:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 15/37 2021-07-10T09:16:43.654 INFO:teuthology.orchestra.run.smithi062.stdout:(34/37): python3-webtest-2.0.33-1.el8.noarch.rp 3.6 MB/s | 85 kB 00:00 2021-07-10T09:16:43.664 INFO:teuthology.orchestra.run.smithi062.stdout:(35/37): python3-waitress-1.2.1-2.el8.1.noarch. 3.6 MB/s | 241 kB 00:00 2021-07-10T09:16:43.676 INFO:teuthology.orchestra.run.smithi062.stdout:(36/37): python3-zc-lockfile-2.0-2.el8.noarch.r 1.0 MB/s | 23 kB 00:00 2021-07-10T09:16:43.912 INFO:teuthology.orchestra.run.smithi062.stdout:(37/37): ceph-osd-17.0.0-5861.g92adad34.el8.x86 18 MB/s | 17 MB 00:00 2021-07-10T09:16:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:16:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:Total 23 MB/s | 29 MB 00:01 2021-07-10T09:16:43.927 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:16:43.970 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:16:43.970 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:16:44.051 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 16/37 2021-07-10T09:16:44.279 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 17/37 2021-07-10T09:16:44.295 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:16:44.299 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:16:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 18/37 2021-07-10T09:16:44.668 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 19/37 2021-07-10T09:16:44.893 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/37 2021-07-10T09:16:45.151 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 21/37 2021-07-10T09:16:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 22/37 2021-07-10T09:16:45.586 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:16:45.603 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 23/37 2021-07-10T09:16:45.801 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-07-10T09:16:45.840 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/37 2021-07-10T09:16:45.994 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-07-10T09:16:46.011 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 25/37 2021-07-10T09:16:46.148 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/37 2021-07-10T09:16:46.188 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 26/37 2021-07-10T09:16:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/37 2021-07-10T09:16:46.442 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-2.6-1.el8.noarch 27/37 2021-07-10T09:16:46.538 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/37 2021-07-10T09:16:46.693 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/37 2021-07-10T09:16:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 28/37 2021-07-10T09:16:46.817 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/37 2021-07-10T09:16:46.953 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 29/37 2021-07-10T09:16:46.984 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/37 2021-07-10T09:16:47.112 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/37 2021-07-10T09:16:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 30/37 2021-07-10T09:16:47.342 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/37 2021-07-10T09:16:47.580 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 11/37 2021-07-10T09:16:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-07-10T09:16:47.814 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 12/37 2021-07-10T09:16:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-5861.g92adad34.el8. 32/37 2021-07-10T09:16:48.027 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 13/37 2021-07-10T09:16:48.181 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 33/37 2021-07-10T09:16:48.279 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 14/37 2021-07-10T09:16:48.486 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 15/37 2021-07-10T09:16:48.907 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 16/37 2021-07-10T09:16:49.161 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 17/37 2021-07-10T09:16:49.331 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 18/37 2021-07-10T09:16:49.588 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 19/37 2021-07-10T09:16:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 33/37 2021-07-10T09:16:49.791 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/37 2021-07-10T09:16:50.015 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 34/37 2021-07-10T09:16:50.096 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 21/37 2021-07-10T09:16:50.330 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 22/37 2021-07-10T09:16:50.406 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 34/37 2021-07-10T09:16:50.589 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 35/37 2021-07-10T09:16:50.590 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 23/37 2021-07-10T09:16:50.797 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 35/37 2021-07-10T09:16:50.805 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/37 2021-07-10T09:16:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 36/37 2021-07-10T09:16:50.959 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 25/37 2021-07-10T09:16:50.975 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 36/37 2021-07-10T09:16:51.082 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:17.0.0-5861.g92adad34.el8.x86_64 37/37 2021-07-10T09:16:51.153 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 26/37 2021-07-10T09:16:51.407 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-2.6-1.el8.noarch 27/37 2021-07-10T09:16:51.441 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:17.0.0-5861.g92adad34.el8.x86_64 37/37 2021-07-10T09:16:51.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:17.0.0-5861.g92adad34.el8.x86_64 1/37 2021-07-10T09:16:51.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 2/37 2021-07-10T09:16:51.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 3/37 2021-07-10T09:16:51.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 4/37 2021-07-10T09:16:51.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 5/37 2021-07-10T09:16:51.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-5861.g92adad34.el8. 6/37 2021-07-10T09:16:51.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/37 2021-07-10T09:16:51.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/37 2021-07-10T09:16:51.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/37 2021-07-10T09:16:51.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/37 2021-07-10T09:16:51.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/37 2021-07-10T09:16:51.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/37 2021-07-10T09:16:51.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 13/37 2021-07-10T09:16:51.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 14/37 2021-07-10T09:16:51.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-idna-2.5-5.el8.noarch 15/37 2021-07-10T09:16:51.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 16/37 2021-07-10T09:16:51.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 17/37 2021-07-10T09:16:51.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 18/37 2021-07-10T09:16:51.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 19/37 2021-07-10T09:16:51.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/37 2021-07-10T09:16:51.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/37 2021-07-10T09:16:51.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 22/37 2021-07-10T09:16:51.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 23/37 2021-07-10T09:16:51.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 24/37 2021-07-10T09:16:51.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 25/37 2021-07-10T09:16:51.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 26/37 2021-07-10T09:16:51.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 27/37 2021-07-10T09:16:51.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 28/37 2021-07-10T09:16:51.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-2.6-1.el8.noarch 29/37 2021-07-10T09:16:51.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 30/37 2021-07-10T09:16:51.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 31/37 2021-07-10T09:16:51.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 32/37 2021-07-10T09:16:51.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 33/37 2021-07-10T09:16:51.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/37 2021-07-10T09:16:51.450 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/37 2021-07-10T09:16:51.450 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/37 2021-07-10T09:16:51.660 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 28/37 2021-07-10T09:16:51.882 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 29/37 2021-07-10T09:16:52.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 37/37 2021-07-10T09:16:52.233 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:52.234 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:16:52.234 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:52.234 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:52.234 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:52.235 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:16:52.235 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:52.235 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:52.235 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-07-10T09:16:52.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-07-10T09:16:52.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-07-10T09:16:52.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-07-10T09:16:52.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-07-10T09:16:52.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-07-10T09:16:52.237 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-07-10T09:16:52.237 INFO:teuthology.orchestra.run.smithi105.stdout: python3-idna-2.5-5.el8.noarch 2021-07-10T09:16:52.237 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-6.2-6.el8.noarch 2021-07-10T09:16:52.237 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-07-10T09:16:52.238 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-07-10T09:16:52.238 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.0.6-13.el8.noarch 2021-07-10T09:16:52.238 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-07-10T09:16:52.238 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-07-10T09:16:52.238 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-07-10T09:16:52.239 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-2.6-1.el8.noarch 2021-07-10T09:16:52.239 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-07-10T09:16:52.239 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.14-14.el8.noarch 2021-07-10T09:16:52.239 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-07-10T09:16:52.240 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2017.2-9.el8.noarch 2021-07-10T09:16:52.240 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-07-10T09:16:52.241 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-07-10T09:16:52.241 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-07-10T09:16:52.241 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-07-10T09:16:52.241 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-07-10T09:16:52.242 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-07-10T09:16:52.242 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-07-10T09:16:52.242 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-07-10T09:16:52.242 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-07-10T09:16:52.243 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-07-10T09:16:52.243 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-07-10T09:16:52.243 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:52.243 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:16:52.283 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 30/37 2021-07-10T09:16:52.445 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2021-07-10T09:16:52.642 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-07-10T09:16:52.918 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:04 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:53.047 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-5861.g92adad34.el8. 32/37 2021-07-10T09:16:53.267 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 33/37 2021-07-10T09:16:53.485 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:16:53.666 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:16:53.667 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:16:53.667 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:16:53.738 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2021-07-10T09:16:54.207 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:06 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:54.963 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:16:54.964 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:54.964 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:16:54.964 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:54.964 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:16:54.965 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 68 k 2021-07-10T09:16:54.965 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:54.965 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:16:54.965 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:54.965 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2021-07-10T09:16:54.966 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:54.966 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 68 k 2021-07-10T09:16:54.966 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 298 k 2021-07-10T09:16:54.966 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:16:54.970 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 33/37 2021-07-10T09:16:55.110 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-17.0.0-5861.g92adad34.el8.noarch.rpm 462 kB/s | 68 kB 00:00 2021-07-10T09:16:55.111 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:16:55.112 INFO:teuthology.orchestra.run.smithi105.stdout:Total 458 kB/s | 68 kB 00:00 2021-07-10T09:16:55.112 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:16:55.118 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:16:55.118 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:16:55.120 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:16:55.123 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:16:55.168 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:16:55.220 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 34/37 2021-07-10T09:16:55.636 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 34/37 2021-07-10T09:16:55.908 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 35/37 2021-07-10T09:16:56.127 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 35/37 2021-07-10T09:16:56.305 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 36/37 2021-07-10T09:16:56.323 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 36/37 2021-07-10T09:16:56.462 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:17.0.0-5861.g92adad34.el8.x86_64 37/37 2021-07-10T09:16:56.884 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-2:17.0.0-5861.g92adad34.el8.x86_64 37/37 2021-07-10T09:16:56.884 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:17.0.0-5861.g92adad34.el8.x86_64 1/37 2021-07-10T09:16:56.884 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 2/37 2021-07-10T09:16:56.885 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 3/37 2021-07-10T09:16:56.885 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 4/37 2021-07-10T09:16:56.885 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 5/37 2021-07-10T09:16:56.885 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-5861.g92adad34.el8. 6/37 2021-07-10T09:16:56.886 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/37 2021-07-10T09:16:56.886 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/37 2021-07-10T09:16:56.886 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/37 2021-07-10T09:16:56.886 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/37 2021-07-10T09:16:56.887 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/37 2021-07-10T09:16:56.887 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/37 2021-07-10T09:16:56.887 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 13/37 2021-07-10T09:16:56.888 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 14/37 2021-07-10T09:16:56.888 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-idna-2.5-5.el8.noarch 15/37 2021-07-10T09:16:56.888 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 16/37 2021-07-10T09:16:56.888 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 17/37 2021-07-10T09:16:56.889 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 18/37 2021-07-10T09:16:56.889 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 19/37 2021-07-10T09:16:56.889 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/37 2021-07-10T09:16:56.889 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/37 2021-07-10T09:16:56.890 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 22/37 2021-07-10T09:16:56.890 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 23/37 2021-07-10T09:16:56.890 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 24/37 2021-07-10T09:16:56.890 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 25/37 2021-07-10T09:16:56.891 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 26/37 2021-07-10T09:16:56.891 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 27/37 2021-07-10T09:16:56.892 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 28/37 2021-07-10T09:16:56.893 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-2.6-1.el8.noarch 29/37 2021-07-10T09:16:56.893 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 30/37 2021-07-10T09:16:56.893 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 31/37 2021-07-10T09:16:56.893 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 32/37 2021-07-10T09:16:56.893 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 33/37 2021-07-10T09:16:56.894 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/37 2021-07-10T09:16:56.894 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/37 2021-07-10T09:16:56.894 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/37 2021-07-10T09:16:57.340 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:16:57.546 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:16:57.737 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 37/37 2021-07-10T09:16:57.737 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:57.737 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:16:57.738 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:57.738 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:57.738 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:57.738 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:16:57.739 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:57.739 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:16:57.739 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-07-10T09:16:57.739 INFO:teuthology.orchestra.run.smithi062.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-07-10T09:16:57.739 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-07-10T09:16:57.740 INFO:teuthology.orchestra.run.smithi062.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-07-10T09:16:57.740 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-07-10T09:16:57.740 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-07-10T09:16:57.740 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-07-10T09:16:57.741 INFO:teuthology.orchestra.run.smithi062.stdout: python3-idna-2.5-5.el8.noarch 2021-07-10T09:16:57.741 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-6.2-6.el8.noarch 2021-07-10T09:16:57.741 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-07-10T09:16:57.741 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-07-10T09:16:57.741 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.0.6-13.el8.noarch 2021-07-10T09:16:57.742 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-07-10T09:16:57.742 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-07-10T09:16:57.742 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-07-10T09:16:57.742 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-2.6-1.el8.noarch 2021-07-10T09:16:57.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-07-10T09:16:57.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.14-14.el8.noarch 2021-07-10T09:16:57.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-07-10T09:16:57.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2017.2-9.el8.noarch 2021-07-10T09:16:57.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-07-10T09:16:57.744 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-07-10T09:16:57.744 INFO:teuthology.orchestra.run.smithi062.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-07-10T09:16:57.744 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-07-10T09:16:57.744 INFO:teuthology.orchestra.run.smithi062.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-07-10T09:16:57.745 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-07-10T09:16:57.745 INFO:teuthology.orchestra.run.smithi062.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-07-10T09:16:57.745 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-07-10T09:16:57.745 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-07-10T09:16:57.746 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-07-10T09:16:57.746 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-07-10T09:16:57.746 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:16:57.746 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:16:57.747 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:16:57.927 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-base 2021-07-10T09:16:58.423 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:10 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:58.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:16:58.436 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:58.436 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:16:58.436 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:16:58.437 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:58.438 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:16:58.597 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2021-07-10T09:16:59.065 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-base-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:16:59.082 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:11 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:59.254 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:16:59.255 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:16:59.255 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:16:59.317 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install cephadm 2021-07-10T09:16:59.806 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:11 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:16:59.832 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:16:59.833 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:59.833 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2021-07-10T09:16:59.834 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:59.834 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:16:59.835 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-5861.g92adad34.el8 ceph 166 k 2021-07-10T09:16:59.835 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:59.835 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:16:59.835 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:16:59.835 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2021-07-10T09:16:59.836 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:16:59.836 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 166 k 2021-07-10T09:16:59.836 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 519 k 2021-07-10T09:16:59.836 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:00.145 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-17.0.0-5861.g92adad 535 kB/s | 166 kB 00:00 2021-07-10T09:17:00.145 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:00.146 INFO:teuthology.orchestra.run.smithi105.stdout:Total 533 kB/s | 166 kB 00:00 2021-07-10T09:17:00.146 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:00.158 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:00.158 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:00.205 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:00.208 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:00.324 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-5861.g92adad34. 1/1 2021-07-10T09:17:00.611 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:00.612 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:17:00.613 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:00.613 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 68 k 2021-07-10T09:17:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:00.614 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:00.614 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:00.614 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2021-07-10T09:17:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:00.614 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 68 k 2021-07-10T09:17:00.615 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 298 k 2021-07-10T09:17:00.615 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-5861.g92adad34. 1/1 2021-07-10T09:17:00.957 INFO:teuthology.orchestra.run.smithi062.stdout:cephadm-17.0.0-5861.g92adad34.el8.noarch.rpm 196 kB/s | 68 kB 00:00 2021-07-10T09:17:00.959 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:00.959 INFO:teuthology.orchestra.run.smithi062.stdout:Total 195 kB/s | 68 kB 00:00 2021-07-10T09:17:00.959 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:00.967 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:00.967 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:00.969 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:00.972 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-5861.g92adad34. 1/1 2021-07-10T09:17:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:01.244 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:01.245 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:01.440 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2021-07-10T09:17:01.910 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:13 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:02.509 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:02.690 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:02.691 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:02.691 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:02.758 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2021-07-10T09:17:03.220 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:15 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:03.830 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:17:04.078 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:17:04.279 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:17:04.568 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/1 2021-07-10T09:17:04.568 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:04.568 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:04.569 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:04.569 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:04.569 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:04.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-immutable-object-cache 2021-07-10T09:17:04.991 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:04.992 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:17:04.993 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:04.993 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 2.3 M 2021-07-10T09:17:04.993 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:17:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 23 k 2021-07-10T09:17:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 8.9 k 2021-07-10T09:17:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-07-10T09:17:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-07-10T09:17:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-07-10T09:17:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:04.995 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:04.995 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:04.995 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2021-07-10T09:17:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:04.996 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.6 M 2021-07-10T09:17:04.997 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 15 M 2021-07-10T09:17:04.997 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): ceph-prometheus-alerts-17.0.0-5861.g92ad 49 kB/s | 8.9 kB 00:00 2021-07-10T09:17:05.215 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): ceph-grafana-dashboards-17.0.0-5861.g92a 106 kB/s | 23 kB 00:00 2021-07-10T09:17:05.231 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 811 kB/s | 43 kB 00:00 2021-07-10T09:17:05.252 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:17 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:05.297 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 410 kB/s | 34 kB 00:00 2021-07-10T09:17:05.391 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2021-07-10T09:17:05.851 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): ceph-mgr-dashboard-17.0.0-5861.g92adad34 2.7 MB/s | 2.3 MB 00:00 2021-07-10T09:17:05.854 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:05.854 INFO:teuthology.orchestra.run.smithi105.stdout:Total 3.0 MB/s | 2.6 MB 00:00 2021-07-10T09:17:05.856 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:05.865 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:05.866 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:05.993 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:05.997 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:06.042 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:06.043 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:06.044 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2021-07-10T09:17:06.044 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:06.045 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:06.045 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-5861.g92adad34.el8 ceph 166 k 2021-07-10T09:17:06.045 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:06.046 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:06.046 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:06.046 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2021-07-10T09:17:06.047 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:06.047 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 166 k 2021-07-10T09:17:06.047 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 519 k 2021-07-10T09:17:06.048 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:06.221 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-immutable-object-cache-17.0.0-5861.g92adad 941 kB/s | 166 kB 00:00 2021-07-10T09:17:06.222 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:06.222 INFO:teuthology.orchestra.run.smithi062.stdout:Total 934 kB/s | 166 kB 00:00 2021-07-10T09:17:06.223 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:06.235 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:06.235 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:06.287 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:06.290 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:06.300 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:06.420 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:06.521 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-07-10T09:17:06.740 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-07-10T09:17:06.932 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-07-10T09:17:07.063 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-5861.g92adad34.el8.n 4/6 2021-07-10T09:17:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-5861.g92adad34. 1/1 2021-07-10T09:17:07.957 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-5861.g92adad34. 1/1 2021-07-10T09:17:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-5861.g92adad34.el8. 5/6 2021-07-10T09:17:08.310 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-5861.g92adad34. 1/1 2021-07-10T09:17:08.310 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:08.311 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:08.311 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:08.311 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:08.311 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:08.386 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarc 6/6 2021-07-10T09:17:08.498 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr 2021-07-10T09:17:08.622 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarc 6/6 2021-07-10T09:17:08.623 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-5861.g92adad34.el8. 1/6 2021-07-10T09:17:08.623 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarc 2/6 2021-07-10T09:17:08.623 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-5861.g92adad34.el8.n 3/6 2021-07-10T09:17:08.624 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-07-10T09:17:08.624 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-07-10T09:17:08.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-07-10T09:17:08.925 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:08.926 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:08.926 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:08.926 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:08.926 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:08.927 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-07-10T09:17:08.927 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-07-10T09:17:08.927 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.4.1-12.el8.noarch 2021-07-10T09:17:08.927 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:08.927 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:08.993 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:20 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:09.023 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2021-07-10T09:17:09.486 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:21 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:09.639 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-mgr-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:09.829 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:09.830 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:09.831 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:09.890 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-dashboard 2021-07-10T09:17:10.275 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:10.276 INFO:teuthology.orchestra.run.smithi105.stdout:===================================================================================================== 2021-07-10T09:17:10.276 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:17:10.277 INFO:teuthology.orchestra.run.smithi105.stdout:===================================================================================================== 2021-07-10T09:17:10.277 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:10.277 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 7.4 M 2021-07-10T09:17:10.277 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:17:10.277 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 8.5.0-2.el8 CentOS-Base 643 k 2021-07-10T09:17:10.278 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 8.5.0-2.el8 CentOS-Base 170 k 2021-07-10T09:17:10.278 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-07-10T09:17:10.278 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-07-10T09:17:10.278 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-07-10T09:17:10.279 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-07-10T09:17:10.279 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-07-10T09:17:10.279 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:10.279 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:10.280 INFO:teuthology.orchestra.run.smithi105.stdout:===================================================================================================== 2021-07-10T09:17:10.280 INFO:teuthology.orchestra.run.smithi105.stdout:Install 8 Packages 2021-07-10T09:17:10.280 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:10.281 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 35 M 2021-07-10T09:17:10.281 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 194 M 2021-07-10T09:17:10.281 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:10.378 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:22 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:10.388 INFO:teuthology.orchestra.run.smithi105.stdout:(1/8): openblas-0.3.12-1.el8.x86_64.rpm 42 MB/s | 4.6 MB 00:00 2021-07-10T09:17:10.459 INFO:teuthology.orchestra.run.smithi105.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 53 MB/s | 3.7 MB 00:00 2021-07-10T09:17:10.465 INFO:teuthology.orchestra.run.smithi105.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 38 MB/s | 225 kB 00:00 2021-07-10T09:17:10.483 INFO:teuthology.orchestra.run.smithi105.stdout:(4/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2021-07-10T09:17:10.509 INFO:teuthology.orchestra.run.smithi105.stdout:(5/8): libgfortran-8.5.0-2.el8.x86_64.rpm 25 MB/s | 643 kB 00:00 2021-07-10T09:17:10.516 INFO:teuthology.orchestra.run.smithi105.stdout:(6/8): libquadmath-8.5.0-2.el8.x86_64.rpm 23 MB/s | 170 kB 00:00 2021-07-10T09:17:10.734 INFO:teuthology.orchestra.run.smithi105.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 51 MB/s | 14 MB 00:00 2021-07-10T09:17:11.174 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:11.175 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:11.175 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:17:11.176 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:11.176 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:11.176 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 2.3 M 2021-07-10T09:17:11.177 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:17:11.177 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 23 k 2021-07-10T09:17:11.177 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 8.9 k 2021-07-10T09:17:11.178 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-07-10T09:17:11.178 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-07-10T09:17:11.178 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-07-10T09:17:11.179 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:11.179 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:11.179 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:11.180 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2021-07-10T09:17:11.180 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:11.181 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 2.6 M 2021-07-10T09:17:11.181 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 15 M 2021-07-10T09:17:11.182 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:11.291 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): ceph-prometheus-alerts-17.0.0-5861.g92ad 79 kB/s | 8.9 kB 00:00 2021-07-10T09:17:11.305 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): ceph-grafana-dashboards-17.0.0-5861.g92a 187 kB/s | 23 kB 00:00 2021-07-10T09:17:11.321 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-07-10T09:17:11.376 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 495 kB/s | 34 kB 00:00 2021-07-10T09:17:11.429 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.8 MB/s | 196 kB 00:00 2021-07-10T09:17:11.527 INFO:teuthology.orchestra.run.smithi105.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-586 5.9 MB/s | 7.4 MB 00:01 2021-07-10T09:17:11.531 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:11.531 INFO:teuthology.orchestra.run.smithi105.stdout:Total 28 MB/s | 35 MB 00:01 2021-07-10T09:17:11.539 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:11.546 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): ceph-mgr-dashboard-17.0.0-5861.g92adad34 6.2 MB/s | 2.3 MB 00:00 2021-07-10T09:17:11.548 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:11.549 INFO:teuthology.orchestra.run.smithi062.stdout:Total 6.9 MB/s | 2.6 MB 00:00 2021-07-10T09:17:11.550 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:11.558 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:11.558 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:11.560 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:11.560 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:11.688 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:11.691 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:11.862 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:11.865 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:11.996 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:12.229 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-07-10T09:17:12.412 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:12.482 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-07-10T09:17:12.571 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-8.5.0-2.el8.x86_64 1/8 2021-07-10T09:17:12.688 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-07-10T09:17:12.709 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libquadmath-8.5.0-2.el8.x86_64 1/8 2021-07-10T09:17:12.879 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-8.5.0-2.el8.x86_64 2/8 2021-07-10T09:17:13.392 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libgfortran-8.5.0-2.el8.x86_64 2/8 2021-07-10T09:17:13.546 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-07-10T09:17:13.953 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-5861.g92adad34.el8.n 4/6 2021-07-10T09:17:14.078 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-07-10T09:17:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-07-10T09:17:14.931 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-07-10T09:17:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-5861.g92adad34.el8. 5/6 2021-07-10T09:17:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-07-10T09:17:15.313 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarc 6/6 2021-07-10T09:17:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-10T09:17:15.576 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarc 6/6 2021-07-10T09:17:15.578 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-5861.g92adad34.el8. 1/6 2021-07-10T09:17:15.578 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarc 2/6 2021-07-10T09:17:15.579 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-5861.g92adad34.el8.n 3/6 2021-07-10T09:17:15.579 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-07-10T09:17:15.579 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-07-10T09:17:15.907 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-07-10T09:17:15.907 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:15.908 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:15.908 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:15.909 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:15.909 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:15.909 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-07-10T09:17:15.909 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-07-10T09:17:15.910 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.4.1-12.el8.noarch 2021-07-10T09:17:15.910 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:15.910 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:15.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-diskprediction-local 2021-07-10T09:17:16.490 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:28 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:17.284 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-10T09:17:17.336 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:17.337 INFO:teuthology.orchestra.run.smithi062.stdout:===================================================================================================== 2021-07-10T09:17:17.337 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:17:17.337 INFO:teuthology.orchestra.run.smithi062.stdout:===================================================================================================== 2021-07-10T09:17:17.338 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:17.338 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 7.4 M 2021-07-10T09:17:17.338 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:17:17.338 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 8.5.0-2.el8 CentOS-Base 643 k 2021-07-10T09:17:17.338 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 8.5.0-2.el8 CentOS-Base 170 k 2021-07-10T09:17:17.339 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-07-10T09:17:17.339 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-07-10T09:17:17.339 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-07-10T09:17:17.339 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-07-10T09:17:17.340 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-07-10T09:17:17.340 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:17.340 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:17.340 INFO:teuthology.orchestra.run.smithi062.stdout:===================================================================================================== 2021-07-10T09:17:17.340 INFO:teuthology.orchestra.run.smithi062.stdout:Install 8 Packages 2021-07-10T09:17:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:17.342 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 35 M 2021-07-10T09:17:17.343 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 194 M 2021-07-10T09:17:17.343 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:17.458 INFO:teuthology.orchestra.run.smithi062.stdout:(1/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 40 MB/s | 4.7 MB 00:00 2021-07-10T09:17:17.501 INFO:teuthology.orchestra.run.smithi062.stdout:(2/8): openblas-0.3.12-1.el8.x86_64.rpm 29 MB/s | 4.6 MB 00:00 2021-07-10T09:17:17.510 INFO:teuthology.orchestra.run.smithi062.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 25 MB/s | 225 kB 00:00 2021-07-10T09:17:17.531 INFO:teuthology.orchestra.run.smithi062.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 52 MB/s | 3.7 MB 00:00 2021-07-10T09:17:17.544 INFO:teuthology.orchestra.run.smithi062.stdout:(5/8): libgfortran-8.5.0-2.el8.x86_64.rpm 49 MB/s | 643 kB 00:00 2021-07-10T09:17:17.553 INFO:teuthology.orchestra.run.smithi062.stdout:(6/8): libquadmath-8.5.0-2.el8.x86_64.rpm 32 MB/s | 170 kB 00:00 2021-07-10T09:17:17.761 INFO:teuthology.orchestra.run.smithi062.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 55 MB/s | 14 MB 00:00 2021-07-10T09:17:18.019 INFO:teuthology.orchestra.run.smithi062.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-586 11 MB/s | 7.4 MB 00:00 2021-07-10T09:17:18.022 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:18.022 INFO:teuthology.orchestra.run.smithi062.stdout:Total 51 MB/s | 35 MB 00:00 2021-07-10T09:17:18.031 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:18.052 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:18.052 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:18.386 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:18.390 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:18.535 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-07-10T09:17:18.737 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad3 8/8 2021-07-10T09:17:18.939 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:19.031 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad3 8/8 2021-07-10T09:17:19.031 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad3 1/8 2021-07-10T09:17:19.032 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-07-10T09:17:19.032 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-07-10T09:17:19.032 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-07-10T09:17:19.032 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-07-10T09:17:19.033 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-07-10T09:17:19.033 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-8.5.0-2.el8.x86_64 7/8 2021-07-10T09:17:19.084 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-8.5.0-2.el8.x86_64 1/8 2021-07-10T09:17:19.249 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libquadmath-8.5.0-2.el8.x86_64 1/8 2021-07-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-8.5.0-2.el8.x86_64 8/8 2021-07-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:19.348 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-8.5.0-2.el8.x86_64 2021-07-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-8.5.0-2.el8.x86_64 2021-07-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.12-1.el8.x86_64 2021-07-10T09:17:19.349 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-07-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-07-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-07-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-07-10T09:17:19.350 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:19.351 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:19.430 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2021-07-10T09:17:19.698 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-8.5.0-2.el8.x86_64 2/8 2021-07-10T09:17:19.893 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:31 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:20.438 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libgfortran-8.5.0-2.el8.x86_64 2/8 2021-07-10T09:17:20.601 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-07-10T09:17:20.654 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:20.655 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2021-07-10T09:17:20.655 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:17:20.655 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2021-07-10T09:17:20.656 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:20.656 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 41 k 2021-07-10T09:17:20.656 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:17:20.656 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-07-10T09:17:20.657 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-07-10T09:17:20.657 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-07-10T09:17:20.657 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-07-10T09:17:20.657 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-07-10T09:17:20.658 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-07-10T09:17:20.658 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-07-10T09:17:20.658 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-07-10T09:17:20.658 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-07-10T09:17:20.659 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-07-10T09:17:20.659 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-07-10T09:17:20.659 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-07-10T09:17:20.659 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:20.659 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:20.660 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2021-07-10T09:17:20.660 INFO:teuthology.orchestra.run.smithi105.stdout:Install 13 Packages 2021-07-10T09:17:20.660 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:20.661 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 1.9 M 2021-07-10T09:17:20.661 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 25 M 2021-07-10T09:17:20.661 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:20.690 INFO:teuthology.orchestra.run.smithi105.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 845 kB/s | 27 kB 00:00 2021-07-10T09:17:20.693 INFO:teuthology.orchestra.run.smithi105.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 576 kB/s | 20 kB 00:00 2021-07-10T09:17:20.695 INFO:teuthology.orchestra.run.smithi105.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 38 MB/s | 126 kB 00:00 2021-07-10T09:17:20.697 INFO:teuthology.orchestra.run.smithi105.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 38 MB/s | 110 kB 00:00 2021-07-10T09:17:20.697 INFO:teuthology.orchestra.run.smithi105.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 47 MB/s | 155 kB 00:00 2021-07-10T09:17:20.708 INFO:teuthology.orchestra.run.smithi105.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 3.5 MB/s | 43 kB 00:00 2021-07-10T09:17:20.780 INFO:teuthology.orchestra.run.smithi105.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 403 kB/s | 33 kB 00:00 2021-07-10T09:17:20.789 INFO:teuthology.orchestra.run.smithi105.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 202 kB/s | 16 kB 00:00 2021-07-10T09:17:20.806 INFO:teuthology.orchestra.run.smithi105.stdout:(9/13): ceph-mgr-rook-17.0.0-5861.g92adad34.el8 277 kB/s | 41 kB 00:00 2021-07-10T09:17:20.834 INFO:teuthology.orchestra.run.smithi105.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.5 MB/s | 81 kB 00:00 2021-07-10T09:17:20.862 INFO:teuthology.orchestra.run.smithi105.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-07-10T09:17:20.894 INFO:teuthology.orchestra.run.smithi105.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 831 kB/s | 73 kB 00:00 2021-07-10T09:17:20.980 INFO:teuthology.orchestra.run.smithi105.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.1 MB/s | 1.2 MB 00:00 2021-07-10T09:17:20.982 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:20.982 INFO:teuthology.orchestra.run.smithi105.stdout:Total 5.9 MB/s | 1.9 MB 00:00 2021-07-10T09:17:20.987 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:21.001 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:21.001 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:21.077 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:21.080 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:21.183 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-07-10T09:17:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-07-10T09:17:22.123 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-07-10T09:17:22.428 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:22.465 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-07-10T09:17:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-07-10T09:17:22.717 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-10T09:17:22.886 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-07-10T09:17:23.115 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-07-10T09:17:23.327 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-07-10T09:17:23.552 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-07-10T09:17:23.818 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-07-10T09:17:24.085 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-07-10T09:17:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-07-10T09:17:24.652 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-07-10T09:17:25.410 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-07-10T09:17:25.694 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-07-10T09:17:25.910 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-07-10T09:17:26.002 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-07-10T09:17:26.087 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-07-10T09:17:26.151 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad3 8/8 2021-07-10T09:17:26.244 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 13/13 2021-07-10T09:17:26.474 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad3 8/8 2021-07-10T09:17:26.475 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad3 1/8 2021-07-10T09:17:26.475 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-07-10T09:17:26.475 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-07-10T09:17:26.475 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-07-10T09:17:26.476 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-07-10T09:17:26.476 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-07-10T09:17:26.476 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-8.5.0-2.el8.x86_64 7/8 2021-07-10T09:17:26.499 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 13/13 2021-07-10T09:17:26.499 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 1/13 2021-07-10T09:17:26.499 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-07-10T09:17:26.500 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-07-10T09:17:26.500 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-07-10T09:17:26.500 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-07-10T09:17:26.500 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-07-10T09:17:26.501 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-07-10T09:17:26.501 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-07-10T09:17:26.501 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-07-10T09:17:26.501 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-07-10T09:17:26.501 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-07-10T09:17:26.502 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-07-10T09:17:26.808 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-8.5.0-2.el8.x86_64 8/8 2021-07-10T09:17:26.809 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:26.809 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:26.809 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:26.809 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-8.5.0-2.el8.x86_64 2021-07-10T09:17:26.810 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-8.5.0-2.el8.x86_64 2021-07-10T09:17:26.810 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.12-1.el8.x86_64 2021-07-10T09:17:26.810 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-07-10T09:17:26.810 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-07-10T09:17:26.811 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-07-10T09:17:26.811 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-07-10T09:17:26.811 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:26.811 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:26.882 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-07-10T09:17:26.882 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:26.882 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:26.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:26.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-07-10T09:17:26.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-07-10T09:17:26.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-07-10T09:17:26.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-07-10T09:17:26.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-07-10T09:17:26.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-07-10T09:17:26.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-07-10T09:17:26.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-07-10T09:17:26.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-07-10T09:17:26.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-07-10T09:17:26.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-07-10T09:17:26.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-07-10T09:17:26.886 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:26.887 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:26.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-rook 2021-07-10T09:17:26.967 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2021-07-10T09:17:27.379 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:27.430 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:28.194 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:28.195 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:28.195 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:17:28.195 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:28.195 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:28.195 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 89 k 2021-07-10T09:17:28.196 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:17:28.196 INFO:teuthology.orchestra.run.smithi105.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-07-10T09:17:28.196 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.5.1-6.el8 CentOS-AppStream 4.8 M 2021-07-10T09:17:28.196 INFO:teuthology.orchestra.run.smithi105.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-07-10T09:17:28.197 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-07-10T09:17:28.197 INFO:teuthology.orchestra.run.smithi105.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-07-10T09:17:28.197 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:28.197 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:28.198 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:28.198 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2021-07-10T09:17:28.198 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:28.199 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 5.8 M 2021-07-10T09:17:28.199 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 24 M 2021-07-10T09:17:28.199 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:28.234 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:28.236 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2021-07-10T09:17:28.236 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:17:28.236 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2021-07-10T09:17:28.236 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:28.237 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 41 k 2021-07-10T09:17:28.237 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:17:28.237 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-07-10T09:17:28.237 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-07-10T09:17:28.238 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-07-10T09:17:28.238 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-07-10T09:17:28.238 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-07-10T09:17:28.239 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-07-10T09:17:28.239 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-07-10T09:17:28.239 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-07-10T09:17:28.239 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-07-10T09:17:28.239 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-07-10T09:17:28.240 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-07-10T09:17:28.240 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-07-10T09:17:28.240 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:28.240 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:28.241 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2021-07-10T09:17:28.241 INFO:teuthology.orchestra.run.smithi062.stdout:Install 13 Packages 2021-07-10T09:17:28.241 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:28.242 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 1.9 M 2021-07-10T09:17:28.243 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 25 M 2021-07-10T09:17:28.243 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:28.244 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 11 MB/s | 538 kB 00:00 2021-07-10T09:17:28.282 INFO:teuthology.orchestra.run.smithi062.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 463 kB/s | 20 kB 00:00 2021-07-10T09:17:28.285 INFO:teuthology.orchestra.run.smithi062.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 46 MB/s | 126 kB 00:00 2021-07-10T09:17:28.289 INFO:teuthology.orchestra.run.smithi062.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 48 MB/s | 110 kB 00:00 2021-07-10T09:17:28.289 INFO:teuthology.orchestra.run.smithi062.stdout:(4/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 535 kB/s | 27 kB 00:00 2021-07-10T09:17:28.291 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): python3-babel-2.5.1-6.el8.noarch.rpm 51 MB/s | 4.8 MB 00:00 2021-07-10T09:17:28.291 INFO:teuthology.orchestra.run.smithi062.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 21 MB/s | 43 kB 00:00 2021-07-10T09:17:28.292 INFO:teuthology.orchestra.run.smithi062.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 45 MB/s | 155 kB 00:00 2021-07-10T09:17:28.343 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): ceph-mgr-cephadm-17.0.0-5861.g92adad34.e 608 kB/s | 89 kB 00:00 2021-07-10T09:17:28.384 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 151 kB/s | 21 kB 00:00 2021-07-10T09:17:28.385 INFO:teuthology.orchestra.run.smithi062.stdout:(7/13): ceph-mgr-rook-17.0.0-5861.g92adad34.el8 279 kB/s | 41 kB 00:00 2021-07-10T09:17:28.409 INFO:teuthology.orchestra.run.smithi062.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 280 kB/s | 33 kB 00:00 2021-07-10T09:17:28.410 INFO:teuthology.orchestra.run.smithi062.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2021-07-10T09:17:28.456 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2021-07-10T09:17:28.465 INFO:teuthology.orchestra.run.smithi062.stdout:(10/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.3 MB/s | 73 kB 00:00 2021-07-10T09:17:28.493 INFO:teuthology.orchestra.run.smithi062.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-07-10T09:17:28.561 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-remoto-1.1.4-4.el8.noarch.rpm 219 kB/s | 48 kB 00:00 2021-07-10T09:17:28.564 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:28.565 INFO:teuthology.orchestra.run.smithi105.stdout:Total 16 MB/s | 5.8 MB 00:00 2021-07-10T09:17:28.567 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:28.577 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:28.577 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:28.596 INFO:teuthology.orchestra.run.smithi062.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 6.2 MB/s | 1.2 MB 00:00 2021-07-10T09:17:28.620 INFO:teuthology.orchestra.run.smithi062.stdout:(13/13): python3-google-auth-1.1.1-10.el8.noarc 347 kB/s | 81 kB 00:00 2021-07-10T09:17:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:Total 5.0 MB/s | 1.9 MB 00:00 2021-07-10T09:17:28.630 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:28.644 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:28.644 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:28.659 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:28.661 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:28.727 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:28.729 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:28.931 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-07-10T09:17:29.250 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:29.491 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-07-10T09:17:29.492 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-07-10T09:17:29.720 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-07-10T09:17:30.007 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-07-10T09:17:30.192 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-07-10T09:17:30.426 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-07-10T09:17:30.441 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-07-10T09:17:30.672 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-07-10T09:17:30.828 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.5.1-6.el8.noarch 4/6 2021-07-10T09:17:31.075 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-07-10T09:17:31.081 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-07-10T09:17:31.277 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 6/6 2021-07-10T09:17:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-07-10T09:17:31.513 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 6/6 2021-07-10T09:17:31.514 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/6 2021-07-10T09:17:31.514 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.5.1-6.el8.noarch 2/6 2021-07-10T09:17:31.515 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-07-10T09:17:31.515 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-07-10T09:17:31.515 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-07-10T09:17:31.813 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-07-10T09:17:31.813 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:31.813 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:31.813 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:31.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-apipkg-1.5-6.el8.noarch 2021-07-10T09:17:31.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.5.1-6.el8.noarch 2021-07-10T09:17:31.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-07-10T09:17:31.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.10.1-3.el8.noarch 2021-07-10T09:17:31.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-07-10T09:17:31.815 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:31.815 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:31.907 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2021-07-10T09:17:32.372 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:44 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:32.550 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-07-10T09:17:32.800 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-07-10T09:17:33.041 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-07-10T09:17:33.171 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:33.172 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:33.172 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2021-07-10T09:17:33.172 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:33.173 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:33.173 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:17.0.0-5861.g92adad34.el8 ceph 834 k 2021-07-10T09:17:33.173 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2021-07-10T09:17:33.173 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-07-10T09:17:33.173 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:33.174 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:33.174 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:33.174 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2021-07-10T09:17:33.175 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:33.175 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 916 k 2021-07-10T09:17:33.176 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 2.8 M 2021-07-10T09:17:33.176 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:33.210 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-07-10T09:17:33.244 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-07-10T09:17:33.450 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 13/13 2021-07-10T09:17:33.727 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 13/13 2021-07-10T09:17:33.727 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 1/13 2021-07-10T09:17:33.727 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-07-10T09:17:33.728 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-07-10T09:17:33.728 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-07-10T09:17:33.728 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-07-10T09:17:33.729 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-07-10T09:17:33.729 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-07-10T09:17:33.729 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-07-10T09:17:33.729 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-07-10T09:17:33.730 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-07-10T09:17:33.730 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-07-10T09:17:33.730 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-07-10T09:17:34.109 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-07-10T09:17:34.109 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:34.110 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:34.110 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:34.111 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-07-10T09:17:34.111 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-07-10T09:17:34.111 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-07-10T09:17:34.111 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-07-10T09:17:34.112 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-07-10T09:17:34.112 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-07-10T09:17:34.112 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-07-10T09:17:34.112 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-07-10T09:17:34.113 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-07-10T09:17:34.113 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-07-10T09:17:34.113 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-07-10T09:17:34.113 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-07-10T09:17:34.114 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:34.114 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:34.133 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-fuse-17.0.0-5861.g92adad34.el8.x86_ 869 kB/s | 834 kB 00:00 2021-07-10T09:17:34.134 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:34.134 INFO:teuthology.orchestra.run.smithi105.stdout:Total 952 kB/s | 916 kB 00:00 2021-07-10T09:17:34.135 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:34.157 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:34.158 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:34.225 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-cephadm 2021-07-10T09:17:34.239 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:34.242 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:34.690 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-07-10T09:17:34.717 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:34.901 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 2/2 2021-07-10T09:17:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 2/2 2021-07-10T09:17:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/2 2021-07-10T09:17:35.499 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:35.500 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:35.501 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:17:35.501 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:35.501 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:35.501 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:17.0.0-5861.g92adad34.el8 ceph-noarch 89 k 2021-07-10T09:17:35.502 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:17:35.502 INFO:teuthology.orchestra.run.smithi062.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-07-10T09:17:35.502 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.5.1-6.el8 CentOS-AppStream 4.8 M 2021-07-10T09:17:35.502 INFO:teuthology.orchestra.run.smithi062.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-07-10T09:17:35.502 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-07-10T09:17:35.503 INFO:teuthology.orchestra.run.smithi062.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-07-10T09:17:35.503 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:35.503 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:35.503 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:35.504 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2021-07-10T09:17:35.504 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:35.505 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 5.8 M 2021-07-10T09:17:35.505 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 24 M 2021-07-10T09:17:35.505 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:35.564 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 8.7 MB/s | 538 kB 00:00 2021-07-10T09:17:35.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-07-10T09:17:35.596 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:35.596 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:35.597 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-07-10T09:17:35.597 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:35.597 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:35.601 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): python3-babel-2.5.1-6.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2021-07-10T09:17:35.632 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 316 kB/s | 21 kB 00:00 2021-07-10T09:17:35.676 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.0 MB/s | 48 kB 00:00 2021-07-10T09:17:35.734 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.6 MB/s | 355 kB 00:00 2021-07-10T09:17:35.757 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2021-07-10T09:17:35.861 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): ceph-mgr-cephadm-17.0.0-5861.g92adad34.e 249 kB/s | 89 kB 00:00 2021-07-10T09:17:35.865 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:35.866 INFO:teuthology.orchestra.run.smithi062.stdout:Total 16 MB/s | 5.8 MB 00:00 2021-07-10T09:17:35.867 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:35.877 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:35.877 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:35.967 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:35.970 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:36.231 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:48 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:36.392 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:36.635 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-07-10T09:17:36.873 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-07-10T09:17:36.987 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:36.988 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:36.989 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2021-07-10T09:17:36.989 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:36.990 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:36.990 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:17.0.0-5861.g92adad34.el8 ceph 125 k 2021-07-10T09:17:36.991 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:36.991 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:36.992 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:36.992 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2021-07-10T09:17:36.993 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:36.993 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 125 k 2021-07-10T09:17:36.994 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 432 k 2021-07-10T09:17:36.994 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:37.226 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-17.0.0-5861.g92adad34.el8.x86_64 523 kB/s | 125 kB 00:00 2021-07-10T09:17:37.227 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:37.227 INFO:teuthology.orchestra.run.smithi105.stdout:Total 519 kB/s | 125 kB 00:00 2021-07-10T09:17:37.228 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:37.234 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:37.234 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:37.283 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:37.286 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:37.408 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:37.842 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:37.911 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-07-10T09:17:38.136 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:38.137 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:38.137 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:38.137 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:38.137 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:38.138 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:38.294 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.5.1-6.el8.noarch 4/6 2021-07-10T09:17:38.551 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-07-10T09:17:38.758 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 6/6 2021-07-10T09:17:39.001 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2021-07-10T09:17:39.022 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 6/6 2021-07-10T09:17:39.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 1/6 2021-07-10T09:17:39.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.5.1-6.el8.noarch 2/6 2021-07-10T09:17:39.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-07-10T09:17:39.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-07-10T09:17:39.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-07-10T09:17:39.353 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-07-10T09:17:39.353 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:39.354 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:39.354 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:17.0.0-5861.g92adad34.el8.noarch 2021-07-10T09:17:39.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-apipkg-1.5-6.el8.noarch 2021-07-10T09:17:39.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.5.1-6.el8.noarch 2021-07-10T09:17:39.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-07-10T09:17:39.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.10.1-3.el8.noarch 2021-07-10T09:17:39.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-07-10T09:17:39.355 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:39.356 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:39.447 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-fuse 2021-07-10T09:17:39.478 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:51 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:39.940 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:51 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:40.054 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:40.234 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:40.235 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:40.236 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:40.304 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2021-07-10T09:17:40.736 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:40.737 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:40.737 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2021-07-10T09:17:40.737 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:40.738 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:40.738 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:17.0.0-5861.g92adad34.el8 ceph 834 k 2021-07-10T09:17:40.738 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2021-07-10T09:17:40.739 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-07-10T09:17:40.739 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:40.739 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:40.739 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:40.739 INFO:teuthology.orchestra.run.smithi062.stdout:Install 2 Packages 2021-07-10T09:17:40.740 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:40.740 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 916 k 2021-07-10T09:17:40.740 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 2.8 M 2021-07-10T09:17:40.741 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:40.770 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:52 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:40.771 INFO:teuthology.orchestra.run.smithi062.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-07-10T09:17:41.467 INFO:teuthology.orchestra.run.smithi062.stdout:(2/2): ceph-fuse-17.0.0-5861.g92adad34.el8.x86_ 1.1 MB/s | 834 kB 00:00 2021-07-10T09:17:41.468 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:41.468 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.2 MB/s | 916 kB 00:00 2021-07-10T09:17:41.470 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:41.493 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:41.494 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:41.550 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:41.551 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:41.552 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2021-07-10T09:17:41.552 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:41.552 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:41.552 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:17.0.0-5861.g92adad34.el8 ceph 24 k 2021-07-10T09:17:41.553 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:41.553 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:41.553 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:41.553 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2021-07-10T09:17:41.553 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:41.554 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 24 k 2021-07-10T09:17:41.554 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 102 k 2021-07-10T09:17:41.554 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:41.582 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:41.585 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:41.753 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:42.067 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-07-10T09:17:42.341 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 2/2 2021-07-10T09:17:42.743 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 2/2 2021-07-10T09:17:42.744 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/2 2021-07-10T09:17:43.101 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-07-10T09:17:43.102 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:43.102 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:43.102 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-07-10T09:17:43.103 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:43.103 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:43.284 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados-devel 2021-07-10T09:17:43.780 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:55 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:44.577 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:44.578 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:44.578 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2021-07-10T09:17:44.578 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:44.579 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:44.579 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:17.0.0-5861.g92adad34.el8 ceph 125 k 2021-07-10T09:17:44.579 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:44.579 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:44.580 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:44.580 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2021-07-10T09:17:44.580 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:44.580 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 125 k 2021-07-10T09:17:44.581 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 432 k 2021-07-10T09:17:44.581 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:44.747 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-17.0.0-5861.g92adad34.el8.x86_6 7.5 kB/s | 24 kB 00:03 2021-07-10T09:17:44.748 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:44.748 INFO:teuthology.orchestra.run.smithi105.stdout:Total 7.5 kB/s | 24 kB 00:03 2021-07-10T09:17:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:44.749 INFO:teuthology.orchestra.run.smithi062.stdout:librados-devel-17.0.0-5861.g92adad34.el8.x86_64 730 kB/s | 125 kB 00:00 2021-07-10T09:17:44.751 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:44.751 INFO:teuthology.orchestra.run.smithi062.stdout:Total 721 kB/s | 125 kB 00:00 2021-07-10T09:17:44.751 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:44.752 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:44.752 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:44.755 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:44.758 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:44.758 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:44.759 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:44.811 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:44.814 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:44.955 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:45.139 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:45.216 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:45.393 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:45.452 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:45.728 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:45.728 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:45.728 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:45.729 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:45.729 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:45.729 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:45.746 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:45.747 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:45.747 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:45.747 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:45.747 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:45.748 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:45.808 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2021-07-10T09:17:45.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs2 2021-07-10T09:17:46.277 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:58 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:46.400 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:58 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:46.861 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:47.000 INFO:teuthology.orchestra.run.smithi062.stdout:Package libcephfs2-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:47.043 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:47.044 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:47.045 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:47.112 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2021-07-10T09:17:47.190 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:47.191 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:47.192 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:47.253 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs-devel 2021-07-10T09:17:47.577 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:59 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:47.750 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:59 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:48.144 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:48.324 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:48.325 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:48.325 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:48.394 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2021-07-10T09:17:48.539 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:48.540 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:48.540 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2021-07-10T09:17:48.541 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:48.541 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:48.541 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:17.0.0-5861.g92adad34.el8 ceph 24 k 2021-07-10T09:17:48.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:48.541 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:48.542 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:48.542 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2021-07-10T09:17:48.542 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:48.542 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 24 k 2021-07-10T09:17:48.543 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 102 k 2021-07-10T09:17:48.543 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:48.667 INFO:teuthology.orchestra.run.smithi062.stdout:libcephfs-devel-17.0.0-5861.g92adad34.el8.x86_6 190 kB/s | 24 kB 00:00 2021-07-10T09:17:48.668 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:48.668 INFO:teuthology.orchestra.run.smithi062.stdout:Total 188 kB/s | 24 kB 00:00 2021-07-10T09:17:48.668 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:48.671 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:48.672 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:48.674 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:48.677 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:48.731 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:17:48.865 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:00 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:48.913 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:49.175 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:49.452 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:49.453 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:49.453 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:17:49.453 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:49.454 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:49.454 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:49.460 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:49.526 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados2 2021-07-10T09:17:49.640 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:49.641 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:49.642 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:49.704 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2021-07-10T09:17:50.020 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:02 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:50.172 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:02 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:50.666 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:50.856 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:50.857 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:50.858 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:50.918 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librbd1 2021-07-10T09:17:51.227 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:51.408 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:51.409 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:51.409 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:51.410 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:03 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:51.477 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2021-07-10T09:17:51.943 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:03 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:52.159 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:52.348 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:52.350 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:52.351 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:52.552 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:52.731 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:52.732 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:52.732 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:52.789 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rados 2021-07-10T09:17:52.797 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2021-07-10T09:17:53.265 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:05 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:53.290 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:05 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:53.877 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:53.916 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rados-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:54.057 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:54.058 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:17:54.059 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:54.104 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:54.105 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:54.106 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:54.127 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2021-07-10T09:17:54.165 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rgw 2021-07-10T09:17:54.592 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:06 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:54.660 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:06 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:55.290 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rgw-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:55.341 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:55.342 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:55.342 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2021-07-10T09:17:55.343 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:55.343 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:55.343 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:17.0.0-5861.g92adad34.el8 ceph 92 k 2021-07-10T09:17:55.343 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:55.343 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:55.344 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:55.344 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2021-07-10T09:17:55.344 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:55.344 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 92 k 2021-07-10T09:17:55.345 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 264 k 2021-07-10T09:17:55.345 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:55.481 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:55.482 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:55.483 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:55.511 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-17.0.0-5861.g92adad34.el8.x86_64.rpm 548 kB/s | 92 kB 00:00 2021-07-10T09:17:55.512 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:55.513 INFO:teuthology.orchestra.run.smithi105.stdout:Total 543 kB/s | 92 kB 00:00 2021-07-10T09:17:55.513 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:55.523 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:55.523 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:55.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-cephfs 2021-07-10T09:17:55.572 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:55.576 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:55.696 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:55.939 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:56.044 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:08 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:56.206 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:56.674 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-cephfs-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:56.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:56.826 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:56.827 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:56.827 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:56.827 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:56.827 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:17:56.866 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:56.867 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:56.868 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:56.949 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rbd 2021-07-10T09:17:56.964 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2021-07-10T09:17:57.438 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:09 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:57.445 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:09 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:58.040 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rbd-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:17:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:17:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:58.201 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2021-07-10T09:17:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:17:58.201 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.7 M 2021-07-10T09:17:58.202 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:17:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:17:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2021-07-10T09:17:58.203 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.7 M 2021-07-10T09:17:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 17 M 2021-07-10T09:17:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:17:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:17:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:17:58.291 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-fuse 2021-07-10T09:17:58.545 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-17.0.0-5861.g92adad34.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-07-10T09:17:58.546 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:58.546 INFO:teuthology.orchestra.run.smithi105.stdout:Total 11 MB/s | 3.7 MB 00:00 2021-07-10T09:17:58.547 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:17:58.559 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:17:58.560 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:17:58.624 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:17:58.627 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:17:58.791 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:10 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:17:59.087 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:17:59.290 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:59.675 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:59.684 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:17:59.685 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:59.686 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2021-07-10T09:17:59.686 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:59.687 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:17:59.687 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:17.0.0-5861.g92adad34.el8 ceph 92 k 2021-07-10T09:17:59.688 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:59.688 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:17:59.688 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:17:59.688 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2021-07-10T09:17:59.689 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:17:59.689 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 92 k 2021-07-10T09:17:59.689 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 264 k 2021-07-10T09:17:59.689 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:17:59.855 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-fuse-17.0.0-5861.g92adad34.el8.x86_64.rpm 542 kB/s | 92 kB 00:00 2021-07-10T09:17:59.856 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:17:59.857 INFO:teuthology.orchestra.run.smithi062.stdout:Total 538 kB/s | 92 kB 00:00 2021-07-10T09:17:59.857 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:17:59.869 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:17:59.869 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:17:59.924 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:17:59.927 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:17:59.954 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:17:59.954 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:59.955 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:17:59.955 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:17:59.955 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:17:59.955 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:18:00.045 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:18:00.121 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2021-07-10T09:18:00.254 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:00.517 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:00.596 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:12 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:18:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:00.812 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:18:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:18:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:00.812 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:18:00.967 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-mirror 2021-07-10T09:18:01.351 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:18:01.352 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:18:01.352 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2021-07-10T09:18:01.352 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:18:01.352 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2021-07-10T09:18:01.353 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:17.0.0-5861.g92adad34.el8 ceph 171 k 2021-07-10T09:18:01.353 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:01.353 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2021-07-10T09:18:01.353 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2021-07-10T09:18:01.354 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2021-07-10T09:18:01.354 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:01.354 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 171 k 2021-07-10T09:18:01.354 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 539 k 2021-07-10T09:18:01.355 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2021-07-10T09:18:01.467 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:13 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:18:01.522 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-17.0.0-5861.g92adad34.el8.x86_64.rpm 1.0 MB/s | 171 kB 00:00 2021-07-10T09:18:01.523 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:18:01.524 INFO:teuthology.orchestra.run.smithi105.stdout:Total 993 kB/s | 171 kB 00:00 2021-07-10T09:18:01.524 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2021-07-10T09:18:01.536 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2021-07-10T09:18:01.536 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2021-07-10T09:18:01.588 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2021-07-10T09:18:01.590 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2021-07-10T09:18:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2021-07-10T09:18:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:02.266 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:18:02.268 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:18:02.268 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2021-07-10T09:18:02.268 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:18:02.269 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:18:02.269 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:17.0.0-5861.g92adad34.el8 ceph 3.7 M 2021-07-10T09:18:02.269 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:02.269 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:18:02.270 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:18:02.270 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2021-07-10T09:18:02.271 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:02.271 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 3.7 M 2021-07-10T09:18:02.271 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 17 M 2021-07-10T09:18:02.271 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:18:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:02.612 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-mirror-17.0.0-5861.g92adad34.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-07-10T09:18:02.613 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:18:02.614 INFO:teuthology.orchestra.run.smithi062.stdout:Total 11 MB/s | 3.7 MB 00:00 2021-07-10T09:18:02.614 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:18:02.628 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:18:02.628 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:18:02.698 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:18:02.701 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:18:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:02.734 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2021-07-10T09:18:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:18:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:02.735 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:18:02.908 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2021-07-10T09:18:03.207 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:18:03.380 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:15 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:18:03.429 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:03.838 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:03.975 INFO:teuthology.orchestra.run.smithi105.stdout:Package rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:18:04.154 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2021-07-10T09:18:04.157 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2021-07-10T09:18:04.158 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2021-07-10T09:18:04.177 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:04.178 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:04.178 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:18:04.178 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:18:04.179 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:04.179 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:18:04.227 DEBUG:teuthology.parallel:result is None 2021-07-10T09:18:04.363 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-nbd 2021-07-10T09:18:04.861 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:16 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:18:05.696 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:18:05.697 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:18:05.697 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2021-07-10T09:18:05.698 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:18:05.698 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2021-07-10T09:18:05.699 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:17.0.0-5861.g92adad34.el8 ceph 171 k 2021-07-10T09:18:05.699 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:05.699 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2021-07-10T09:18:05.699 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2021-07-10T09:18:05.700 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2021-07-10T09:18:05.700 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:05.700 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 171 k 2021-07-10T09:18:05.700 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 539 k 2021-07-10T09:18:05.701 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2021-07-10T09:18:05.869 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-nbd-17.0.0-5861.g92adad34.el8.x86_64.rpm 997 kB/s | 171 kB 00:00 2021-07-10T09:18:05.870 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2021-07-10T09:18:05.870 INFO:teuthology.orchestra.run.smithi062.stdout:Total 989 kB/s | 171 kB 00:00 2021-07-10T09:18:05.870 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2021-07-10T09:18:05.884 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2021-07-10T09:18:05.884 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2021-07-10T09:18:05.938 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2021-07-10T09:18:05.942 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2021-07-10T09:18:06.087 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2021-07-10T09:18:06.286 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:06.556 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:06.912 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 1/1 2021-07-10T09:18:06.912 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:06.913 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2021-07-10T09:18:06.913 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:17.0.0-5861.g92adad34.el8.x86_64 2021-07-10T09:18:06.914 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:06.914 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:18:07.052 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-mirror 2021-07-10T09:18:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:19 ago on Sat 10 Jul 2021 09:15:48 AM UTC. 2021-07-10T09:18:08.175 INFO:teuthology.orchestra.run.smithi062.stdout:Package rbd-mirror-2:17.0.0-5861.g92adad34.el8.x86_64 is already installed. 2021-07-10T09:18:08.363 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2021-07-10T09:18:08.364 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2021-07-10T09:18:08.364 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2021-07-10T09:18:08.429 DEBUG:teuthology.parallel:result is None 2021-07-10T09:18:08.430 DEBUG:teuthology.orchestra.run.smithi062:> sudo lsb_release -is 2021-07-10T09:18:08.466 INFO:teuthology.orchestra.run.smithi062.stdout:CentOSStream 2021-07-10T09:18:08.468 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-07-10T09:18:08.476 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=92adad3411b7280fb159eb9d7304c654695beb8b 2021-07-10T09:18:08.650 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-07-10T09:18:08.679 INFO:teuthology.orchestra.run.smithi062.stdout:17.0.0-5861.g92adad34.el8 2021-07-10T09:18:08.680 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-5861.g92adad34.el8 2021-07-10T09:18:08.680 INFO:teuthology.task.install:The correct ceph version 17.0.0-5861.g92adad34 is installed. 2021-07-10T09:18:08.680 DEBUG:teuthology.orchestra.run.smithi105:> sudo lsb_release -is 2021-07-10T09:18:08.722 INFO:teuthology.orchestra.run.smithi105.stdout:CentOSStream 2021-07-10T09:18:08.724 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-07-10T09:18:08.725 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=92adad3411b7280fb159eb9d7304c654695beb8b 2021-07-10T09:18:08.890 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-07-10T09:18:08.914 INFO:teuthology.orchestra.run.smithi105.stdout:17.0.0-5861.g92adad34.el8 2021-07-10T09:18:08.914 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-5861.g92adad34.el8 2021-07-10T09:18:08.915 INFO:teuthology.task.install:The correct ceph version 17.0.0-5861.g92adad34 is installed. 2021-07-10T09:18:08.915 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-07-10T09:18:08.916 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:08.916 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-07-10T09:18:08.943 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:08.944 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-07-10T09:18:08.985 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-07-10T09:18:08.985 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:08.986 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2021-07-10T09:18:09.013 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-07-10T09:18:09.079 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:09.080 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2021-07-10T09:18:09.110 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-07-10T09:18:09.179 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-07-10T09:18:09.180 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:09.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2021-07-10T09:18:09.207 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-07-10T09:18:09.277 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:09.277 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2021-07-10T09:18:09.306 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-07-10T09:18:09.375 INFO:teuthology.run_tasks:Running task ceph... 2021-07-10T09:18:09.444 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-07-10T09:18:09.445 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2021-07-10T09:18:09.447 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /var/log/ceph 2021-07-10T09:18:09.476 INFO:tasks.ceph:Disabling ceph logrotate... 2021-07-10T09:18:09.476 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2021-07-10T09:18:09.514 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/logrotate.d/ceph 2021-07-10T09:18:09.543 INFO:tasks.ceph:Creating extra log directories... 2021-07-10T09:18:09.543 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-07-10T09:18:09.581 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-07-10T09:18:09.624 INFO:tasks.ceph:Creating ceph cluster cluster1... 2021-07-10T09:18:09.625 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-07-10T09:18:09.625 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi/6262049', 'branch': 'wip-mgolub-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'to.my.trociny@gmail.com', 'first_in_suite': False, 'job_id': '6262049', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-mgolub-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mgolub-testing', 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, 'owner': 'scheduled_trociny@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b', 'sleep_before_teardown': 0, 'suite': 'rbd', 'suite_branch': 'wip-mgolub-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '92adad3411b7280fb159eb9d7304c654695beb8b', 'targets': {'smithi062.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjifRisnkBt6E3gXwjqJegeGV3N9KQSLRR3AhOQa1kN790cPaqd8ZBjIE8kHyKSQyoidtLxSE6vJTf07GZRht4joRBX8QVlr98pCqRD+h4zIAlbByupFYseek67lfsGdjW4v8Wrg1ETeERn89rLLhS2njcgQS/ZIRlchOeOuUhjrE+yGbYr7uI0VXp1zbVUsd0OjCyzDJUtOp8bDehlbGPSYS5jmYzfsaH3vDUlWP9xV1GmvyAGJPOXvaaXD2spe9luaLYS55UoQF60M+eSKciQPP0rCNnJqKnpnuRoiuw2hvgrxIwE/TSIbQnBoOqNIKL/dzhhul0TiGitVxelXTZwbdCri2ioPElcJ4YfOru9Lq2+oz9Vpvhr+IuABmzym+5atHmyOII5ueQ9WGTHUoB81A4jfpUBF1DV1vfelvLayrTaFbFyAo7zJwqs5KPy8+/q2egbJtXmnvBF1a5nF/iF+tDfwnGKVSD5/pK9WU79Vi5pmc04F4Gc80yf1haF+0=', 'smithi105.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpGytXlOFlN1ptltJGikwEmNTcon0D67LpimkU8fksWbJap5NMtH0HZq37iy6W2NttOaC/S3NQ2iZ1lyYqX052aPHMgH2IIs70+sUELig4BxOJj6fri+23DBkfXaM+nCQcFlrMvOmJyJiRWaG9MNFXAPOHIMJrBu3CadnSvkkfia/W7f2FxwYqb3Ky7xKf1DUw2lL6UVDYG90t0/aYIPvykDntRlFPBWO1SCQ8hY/ZZzR7ZiXShJ2xT0NiRYrMw1GOJ5cCWKTurL29gr/At8h27Y4hHY7W+3Y9JKap5O1a3rMAcwdrQ7+JEI6QwyxZxE6pzWy8yLbLCp1pdp+9MkF07TJTUCImxVWh/QpYOMYqcxaxCxS/pVorKn3kUi8QPobCLFMRpQGBZTMAXPdDD/ScmifnWnwFWiNedppxdMiF8oJN30gAhqzHg21kki+m2YP4R1fgKleUVKD+3LrYvOy5preWF6hj14c8/80znrRb4O/gKKsU5mchA/6dT2qgv1M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology_branch': 'master', 'teuthology_sha1': '24e3280d7d1fc156628af95b24b27a29e38b2766', 'timestamp': '2021-07-10_06:51:06', 'tube': 'smithi', 'user': 'trociny', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-07-10T09:18:09.626 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2021-07-10T09:18:09.647 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2021-07-10T09:18:09.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2021-07-10T09:18:09.703 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/run/ceph 2021-07-10T09:18:09.730 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:09.730 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2021-07-10T09:18:09.784 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-07-10T09:18:09.784 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-07-10T09:18:09.845 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-07-10T09:18:09.846 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:09.846 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 112657 Links: 1 2021-07-10T09:18:09.847 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:09.848 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:09.848 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2021-07-10 09:18:03.611052486 +0000 2021-07-10T09:18:09.848 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2021-07-10 09:15:14.578814738 +0000 2021-07-10T09:18:09.849 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2021-07-10 09:15:14.578814738 +0000 2021-07-10T09:18:09.849 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2021-07-10T09:18:09.851 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2021-07-10T09:18:09.851 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2021-07-10T09:18:09.851 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000318265 s, 1.6 MB/s 2021-07-10T09:18:09.854 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-07-10T09:18:09.910 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-07-10T09:18:09.911 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:09.911 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 109765 Links: 1 2021-07-10T09:18:09.911 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:09.911 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:09.912 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2021-07-10 09:18:03.611052486 +0000 2021-07-10T09:18:09.912 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2021-07-10 09:15:14.953809022 +0000 2021-07-10T09:18:09.912 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2021-07-10 09:15:14.953809022 +0000 2021-07-10T09:18:09.912 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2021-07-10T09:18:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2021-07-10T09:18:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2021-07-10T09:18:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000187925 s, 2.7 MB/s 2021-07-10T09:18:09.923 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-07-10T09:18:09.979 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-07-10T09:18:09.980 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:09.980 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 107122 Links: 1 2021-07-10T09:18:09.981 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:09.981 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:09.981 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2021-07-10 09:18:03.611052486 +0000 2021-07-10T09:18:09.981 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2021-07-10 09:15:15.335803199 +0000 2021-07-10T09:18:09.981 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2021-07-10 09:15:15.335803199 +0000 2021-07-10T09:18:09.982 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2021-07-10T09:18:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2021-07-10T09:18:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2021-07-10T09:18:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000222358 s, 2.3 MB/s 2021-07-10T09:18:09.992 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-07-10T09:18:10.048 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-07-10T09:18:10.049 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:10.049 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 109945 Links: 1 2021-07-10T09:18:10.049 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:10.049 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:10.050 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2021-07-10 09:18:03.611052486 +0000 2021-07-10T09:18:10.050 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2021-07-10 09:15:15.686797849 +0000 2021-07-10T09:18:10.050 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2021-07-10 09:15:15.686797849 +0000 2021-07-10T09:18:10.050 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2021-07-10T09:18:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2021-07-10T09:18:10.059 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2021-07-10T09:18:10.061 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000196944 s, 2.6 MB/s 2021-07-10T09:18:10.063 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:10.064 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-07-10T09:18:10.064 INFO:tasks.ceph:Generating config... 2021-07-10T09:18:10.065 INFO:tasks.ceph:[client] debug rbd = 20 2021-07-10T09:18:10.066 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2021-07-10T09:18:10.066 INFO:tasks.ceph:[client] log to stderr = False 2021-07-10T09:18:10.067 INFO:tasks.ceph:[client] rbd default features = 125 2021-07-10T09:18:10.067 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2021-07-10T09:18:10.067 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.068 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.068 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.068 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.069 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.069 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.069 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.070 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.071 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-07-10T09:18:10.071 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2021-07-10T09:18:10.071 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-07-10T09:18:10.072 INFO:tasks.ceph:[mgr] debug ms = 1 2021-07-10T09:18:10.072 INFO:tasks.ceph:[mon] debug mon = 20 2021-07-10T09:18:10.072 INFO:tasks.ceph:[mon] debug ms = 1 2021-07-10T09:18:10.073 INFO:tasks.ceph:[mon] debug paxos = 20 2021-07-10T09:18:10.073 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-07-10T09:18:10.074 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2021-07-10T09:18:10.074 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-07-10T09:18:10.074 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-07-10T09:18:10.075 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2021-07-10T09:18:10.075 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2021-07-10T09:18:10.075 INFO:tasks.ceph:[osd] debug ms = 1 2021-07-10T09:18:10.076 INFO:tasks.ceph:[osd] debug osd = 20 2021-07-10T09:18:10.076 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2021-07-10T09:18:10.076 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-07-10T09:18:10.077 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-07-10T09:18:10.077 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-07-10T09:18:10.077 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-07-10T09:18:10.077 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-07-10T09:18:10.078 INFO:tasks.ceph:Setting up cluster1.mon.a... 2021-07-10T09:18:10.078 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2021-07-10T09:18:10.230 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/cluster1.keyring 2021-07-10T09:18:10.233 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2021-07-10T09:18:10.274 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2021-07-10T09:18:10.340 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.62')] 2021-07-10T09:18:10.341 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.62', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2021-07-10T09:18:10.342 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:10.343 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-07-10T09:18:10.397 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.62 --print /home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:10.475 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:10.475 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: generated fsid 7e08b311-22b7-4a7a-a310-0bd6eae2431d 2021-07-10T09:18:10.475 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = octopus 2021-07-10T09:18:10.476 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 0 2021-07-10T09:18:10.476 INFO:teuthology.orchestra.run.smithi062.stdout:fsid 7e08b311-22b7-4a7a-a310-0bd6eae2431d 2021-07-10T09:18:10.476 INFO:teuthology.orchestra.run.smithi062.stdout:last_changed 2021-07-10T09:18:10.474267+0000 2021-07-10T09:18:10.476 INFO:teuthology.orchestra.run.smithi062.stdout:created 2021-07-10T09:18:10.474267+0000 2021-07-10T09:18:10.476 INFO:teuthology.orchestra.run.smithi062.stdout:min_mon_release 15 (octopus) 2021-07-10T09:18:10.477 INFO:teuthology.orchestra.run.smithi062.stdout:election_strategy: 1 2021-07-10T09:18:10.477 INFO:teuthology.orchestra.run.smithi062.stdout:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2021-07-10T09:18:10.477 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2021-07-10T09:18:10.479 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-07-10T09:18:10.536 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 7e08b311-22b7-4a7a-a310-0bd6eae2431d... 2021-07-10T09:18:10.538 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2021-07-10T09:18:10.579 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2021-07-10T09:18:10.621 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2021-07-10T09:18:10.622 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2021-07-10T09:18:10.622 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-07-10T09:18:10.622 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2021-07-10T09:18:10.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.623 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2021-07-10T09:18:10.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.623 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2021-07-10T09:18:10.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2021-07-10T09:18:10.624 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2021-07-10T09:18:10.624 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.624 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2021-07-10T09:18:10.624 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2021-07-10T09:18:10.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2021-07-10T09:18:10.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2021-07-10T09:18:10.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.626 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2021-07-10T09:18:10.626 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2021-07-10T09:18:10.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.626 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2021-07-10T09:18:10.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2021-07-10T09:18:10.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2021-07-10T09:18:10.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2021-07-10T09:18:10.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2021-07-10T09:18:10.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2021-07-10T09:18:10.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2021-07-10T09:18:10.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2021-07-10T09:18:10.629 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-07-10T09:18:10.629 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2021-07-10T09:18:10.629 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2021-07-10T09:18:10.630 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.630 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-07-10T09:18:10.630 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.630 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2021-07-10T09:18:10.631 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.632 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2021-07-10T09:18:10.632 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.632 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2021-07-10T09:18:10.633 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2021-07-10T09:18:10.633 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2021-07-10T09:18:10.633 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = 172.21.15.62 2021-07-10T09:18:10.634 INFO:teuthology.orchestra.run.smithi062.stdout: mon client directed command retry = 5 2021-07-10T09:18:10.634 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject socket failures = 5000 2021-07-10T09:18:10.635 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 7e08b311-22b7-4a7a-a310-0bd6eae2431d 2021-07-10T09:18:10.635 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.635 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2021-07-10T09:18:10.636 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2021-07-10T09:18:10.636 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.636 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2021-07-10T09:18:10.637 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2021-07-10T09:18:10.637 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.637 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2021-07-10T09:18:10.638 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2021-07-10T09:18:10.638 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.638 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2021-07-10T09:18:10.638 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2021-07-10T09:18:10.638 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2021-07-10T09:18:10.639 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.639 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2021-07-10T09:18:10.639 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2021-07-10T09:18:10.639 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.640 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2021-07-10T09:18:10.640 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.640 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2021-07-10T09:18:10.640 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.641 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2021-07-10T09:18:10.641 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.641 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2021-07-10T09:18:10.641 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2021-07-10T09:18:10.641 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.642 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2021-07-10T09:18:10.642 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2021-07-10T09:18:10.642 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2021-07-10T09:18:10.642 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore compression algorithm = snappy 2021-07-10T09:18:10.643 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore compression mode = aggressive 2021-07-10T09:18:10.643 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2021-07-10T09:18:10.643 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2021-07-10T09:18:10.643 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2021-07-10T09:18:10.644 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2021-07-10T09:18:10.644 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2021-07-10T09:18:10.644 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2021-07-10T09:18:10.644 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2021-07-10T09:18:10.644 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2021-07-10T09:18:10.645 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2021-07-10T09:18:10.645 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2021-07-10T09:18:10.645 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2021-07-10T09:18:10.645 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.645 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2021-07-10T09:18:10.646 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2021-07-10T09:18:10.646 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2021-07-10T09:18:10.646 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2021-07-10T09:18:10.646 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2021-07-10T09:18:10.647 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:10.647 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:10.647 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2021-07-10T09:18:10.647 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.648 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2021-07-10T09:18:10.648 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2021-07-10T09:18:10.648 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2021-07-10T09:18:10.648 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2021-07-10T09:18:10.649 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2021-07-10T09:18:10.649 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2021-07-10T09:18:10.649 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2021-07-10T09:18:10.649 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:10.649 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2021-07-10T09:18:10.650 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2021-07-10T09:18:10.650 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:10.650 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.650 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-07-10T09:18:10.651 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2021-07-10T09:18:10.651 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2021-07-10T09:18:10.651 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.651 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2021-07-10T09:18:10.651 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-07-10T09:18:10.652 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-07-10T09:18:10.652 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:10.652 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2021-07-10T09:18:10.652 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2021-07-10T09:18:10.652 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2021-07-10T09:18:10.653 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2021-07-10T09:18:10.653 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-07-10T09:18:10.653 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-07-10T09:18:10.653 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd = 20 2021-07-10T09:18:10.654 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd_mirror = 15 2021-07-10T09:18:10.654 INFO:teuthology.orchestra.run.smithi062.stdout: log to stderr = False 2021-07-10T09:18:10.654 INFO:teuthology.orchestra.run.smithi062.stdout: rbd default features = 125 2021-07-10T09:18:10.654 INFO:teuthology.orchestra.run.smithi062.stdout: rbd mirror image policy type = simple 2021-07-10T09:18:10.654 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2021-07-10T09:18:10.655 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.0] 2021-07-10T09:18:10.655 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.655 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.655 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.1] 2021-07-10T09:18:10.656 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.656 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.656 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.2] 2021-07-10T09:18:10.656 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.657 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.657 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.3] 2021-07-10T09:18:10.657 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.657 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.659 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2021-07-10T09:18:10.659 INFO:teuthology.orchestra.run.smithi105.stdout: chdir = "" 2021-07-10T09:18:10.660 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-07-10T09:18:10.660 INFO:teuthology.orchestra.run.smithi105.stdout: auth supported = cephx 2021-07-10T09:18:10.660 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.660 INFO:teuthology.orchestra.run.smithi105.stdout: filestore xattr use omap = true 2021-07-10T09:18:10.660 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.661 INFO:teuthology.orchestra.run.smithi105.stdout: mon clock drift allowed = 1.000 2021-07-10T09:18:10.661 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.661 INFO:teuthology.orchestra.run.smithi105.stdout: osd crush chooseleaf type = 0 2021-07-10T09:18:10.661 INFO:teuthology.orchestra.run.smithi105.stdout: auth debug = true 2021-07-10T09:18:10.662 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.662 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on old message = true 2021-07-10T09:18:10.662 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on bug = true 2021-07-10T09:18:10.663 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.663 INFO:teuthology.orchestra.run.smithi105.stdout: mon max pg per osd = 10000 # >= luminous 2021-07-10T09:18:10.663 INFO:teuthology.orchestra.run.smithi105.stdout: mon pg warn max object skew = 0 2021-07-10T09:18:10.663 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.663 INFO:teuthology.orchestra.run.smithi105.stdout: # disable pg_autoscaler by default for new pools 2021-07-10T09:18:10.664 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_pg_autoscale_mode = off 2021-07-10T09:18:10.664 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.664 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default size = 2 2021-07-10T09:18:10.664 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.665 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow primary affinity = true 2021-07-10T09:18:10.665 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow pg remap = true 2021-07-10T09:18:10.665 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on legacy crush tunables = false 2021-07-10T09:18:10.665 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on crush straw calc version zero = false 2021-07-10T09:18:10.665 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on no sortbitwise = false 2021-07-10T09:18:10.666 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on osd down out interval zero = false 2021-07-10T09:18:10.666 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on too few osds = false 2021-07-10T09:18:10.666 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-07-10T09:18:10.666 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_no_redundancy = false 2021-07-10T09:18:10.667 INFO:teuthology.orchestra.run.smithi105.stdout: mon_allow_pool_size_one = true 2021-07-10T09:18:10.667 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.667 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-07-10T09:18:10.667 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.668 INFO:teuthology.orchestra.run.smithi105.stdout: osd default data pool replay window = 5 2021-07-10T09:18:10.668 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.668 INFO:teuthology.orchestra.run.smithi105.stdout: mon allow pool delete = true 2021-07-10T09:18:10.668 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.669 INFO:teuthology.orchestra.run.smithi105.stdout: mon cluster log file level = debug 2021-07-10T09:18:10.669 INFO:teuthology.orchestra.run.smithi105.stdout: debug asserts on shutdown = true 2021-07-10T09:18:10.669 INFO:teuthology.orchestra.run.smithi105.stdout: mon health detail to clog = false 2021-07-10T09:18:10.669 INFO:teuthology.orchestra.run.smithi105.stdout: mon host = 172.21.15.62 2021-07-10T09:18:10.669 INFO:teuthology.orchestra.run.smithi105.stdout: mon client directed command retry = 5 2021-07-10T09:18:10.670 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject socket failures = 5000 2021-07-10T09:18:10.670 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = 7e08b311-22b7-4a7a-a310-0bd6eae2431d 2021-07-10T09:18:10.670 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.670 INFO:teuthology.orchestra.run.smithi105.stdout:[osd] 2021-07-10T09:18:10.671 INFO:teuthology.orchestra.run.smithi105.stdout: osd journal size = 100 2021-07-10T09:18:10.671 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.671 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub load threshold = 5.0 2021-07-10T09:18:10.671 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub max interval = 600 2021-07-10T09:18:10.671 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.672 INFO:teuthology.orchestra.run.smithi105.stdout: osd recover clone overlap = true 2021-07-10T09:18:10.672 INFO:teuthology.orchestra.run.smithi105.stdout: osd recovery max chunk = 1048576 2021-07-10T09:18:10.672 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.672 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug shutdown = true 2021-07-10T09:18:10.673 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug op order = true 2021-07-10T09:18:10.673 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify stray on activate = true 2021-07-10T09:18:10.673 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.673 INFO:teuthology.orchestra.run.smithi105.stdout: osd open classes on start = true 2021-07-10T09:18:10.674 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug pg log writeout = true 2021-07-10T09:18:10.674 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.674 INFO:teuthology.orchestra.run.smithi105.stdout: osd deep scrub update digest min age = 30 2021-07-10T09:18:10.674 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.675 INFO:teuthology.orchestra.run.smithi105.stdout: osd map max advance = 10 2021-07-10T09:18:10.675 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.675 INFO:teuthology.orchestra.run.smithi105.stdout: journal zero on create = true 2021-07-10T09:18:10.675 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.676 INFO:teuthology.orchestra.run.smithi105.stdout: filestore ondisk finisher threads = 3 2021-07-10T09:18:10.676 INFO:teuthology.orchestra.run.smithi105.stdout: filestore apply finisher threads = 3 2021-07-10T09:18:10.676 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.677 INFO:teuthology.orchestra.run.smithi105.stdout: bdev debug aio = true 2021-07-10T09:18:10.677 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug misdirected ops = true 2021-07-10T09:18:10.677 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore block size = 96636764160 2021-07-10T09:18:10.677 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression algorithm = snappy 2021-07-10T09:18:10.677 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression mode = aggressive 2021-07-10T09:18:10.678 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore fsck on mount = True 2021-07-10T09:18:10.678 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluefs = 1/20 2021-07-10T09:18:10.678 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluestore = 1/20 2021-07-10T09:18:10.678 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2021-07-10T09:18:10.679 INFO:teuthology.orchestra.run.smithi105.stdout: debug osd = 20 2021-07-10T09:18:10.679 INFO:teuthology.orchestra.run.smithi105.stdout: debug rocksdb = 4/10 2021-07-10T09:18:10.679 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd backfillfull_ratio = 0.85 2021-07-10T09:18:10.679 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd full ratio = 0.9 2021-07-10T09:18:10.679 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd nearfull ratio = 0.8 2021-07-10T09:18:10.680 INFO:teuthology.orchestra.run.smithi105.stdout: osd failsafe full ratio = 0.95 2021-07-10T09:18:10.680 INFO:teuthology.orchestra.run.smithi105.stdout: osd objectstore = bluestore 2021-07-10T09:18:10.680 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.680 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr] 2021-07-10T09:18:10.680 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2021-07-10T09:18:10.681 INFO:teuthology.orchestra.run.smithi105.stdout: debug mgr = 20 2021-07-10T09:18:10.681 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2021-07-10T09:18:10.681 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2021-07-10T09:18:10.681 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:10.682 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:10.682 INFO:teuthology.orchestra.run.smithi105.stdout: mgr/telemetry/nag = false 2021-07-10T09:18:10.682 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.682 INFO:teuthology.orchestra.run.smithi105.stdout:[mon] 2021-07-10T09:18:10.682 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2021-07-10T09:18:10.683 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2021-07-10T09:18:10.683 INFO:teuthology.orchestra.run.smithi105.stdout: debug paxos = 20 2021-07-10T09:18:10.683 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2021-07-10T09:18:10.683 INFO:teuthology.orchestra.run.smithi105.stdout: mon data avail warn = 5 2021-07-10T09:18:10.684 INFO:teuthology.orchestra.run.smithi105.stdout: mon mgr mkfs grace = 240 2021-07-10T09:18:10.684 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:10.684 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd reporter subtree level = osd 2021-07-10T09:18:10.684 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd prime pg temp = true 2021-07-10T09:18:10.685 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:10.685 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.685 INFO:teuthology.orchestra.run.smithi105.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-07-10T09:18:10.685 INFO:teuthology.orchestra.run.smithi105.stdout: auth mon ticket ttl = 660 # 11m 2021-07-10T09:18:10.686 INFO:teuthology.orchestra.run.smithi105.stdout: auth service ticket ttl = 240 # 4m 2021-07-10T09:18:10.686 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.686 INFO:teuthology.orchestra.run.smithi105.stdout: # don't complain about insecure global_id in the test suite 2021-07-10T09:18:10.686 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-07-10T09:18:10.686 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-07-10T09:18:10.687 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:10.687 INFO:teuthology.orchestra.run.smithi105.stdout:[client] 2021-07-10T09:18:10.687 INFO:teuthology.orchestra.run.smithi105.stdout: rgw cache enabled = true 2021-07-10T09:18:10.687 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable ops log = true 2021-07-10T09:18:10.687 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable usage log = true 2021-07-10T09:18:10.688 INFO:teuthology.orchestra.run.smithi105.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-07-10T09:18:10.688 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-07-10T09:18:10.688 INFO:teuthology.orchestra.run.smithi105.stdout: debug rbd = 20 2021-07-10T09:18:10.688 INFO:teuthology.orchestra.run.smithi105.stdout: debug rbd_mirror = 15 2021-07-10T09:18:10.689 INFO:teuthology.orchestra.run.smithi105.stdout: log to stderr = False 2021-07-10T09:18:10.689 INFO:teuthology.orchestra.run.smithi105.stdout: rbd default features = 125 2021-07-10T09:18:10.689 INFO:teuthology.orchestra.run.smithi105.stdout: rbd mirror image policy type = simple 2021-07-10T09:18:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.a] 2021-07-10T09:18:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.0] 2021-07-10T09:18:10.690 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.690 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.690 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.1] 2021-07-10T09:18:10.690 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.691 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.691 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.2] 2021-07-10T09:18:10.691 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.691 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.692 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.3] 2021-07-10T09:18:10.692 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:10.692 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:10.692 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2021-07-10T09:18:10.693 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2021-07-10T09:18:10.752 INFO:tasks.ceph:Copying monmap to all nodes... 2021-07-10T09:18:10.752 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:10.753 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2021-07-10T09:18:10.769 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:10.771 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2021-07-10T09:18:10.824 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:18:10.825 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:10.825 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/cluster1.keyring 2021-07-10T09:18:10.825 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2021-07-10T09:18:10.901 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:10.902 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:10.957 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:18:10.958 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:10.959 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/cluster1.keyring 2021-07-10T09:18:10.959 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2021-07-10T09:18:11.001 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:11.001 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:11.058 INFO:tasks.ceph:Setting up mon nodes... 2021-07-10T09:18:11.058 INFO:tasks.ceph:Setting up mgr nodes... 2021-07-10T09:18:11.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2021-07-10T09:18:11.109 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2021-07-10T09:18:11.111 INFO:tasks.ceph:Setting up mds nodes... 2021-07-10T09:18:11.112 INFO:tasks.ceph_client:Setting up client nodes... 2021-07-10T09:18:11.112 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2021-07-10T09:18:11.152 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/cluster1.client.0.keyring 2021-07-10T09:18:11.165 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2021-07-10T09:18:11.236 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2021-07-10T09:18:11.249 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2021-07-10T09:18:11.290 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2021-07-10T09:18:11.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2021-07-10T09:18:11.386 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2021-07-10T09:18:11.398 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2021-07-10T09:18:11.476 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2021-07-10T09:18:11.488 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2021-07-10T09:18:11.567 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2021-07-10T09:18:11.579 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2021-07-10T09:18:11.657 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2021-07-10T09:18:11.669 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2021-07-10T09:18:11.747 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2021-07-10T09:18:11.759 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2021-07-10T09:18:11.838 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2021-07-10T09:18:11.856 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-07-10T09:18:11.856 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-07-10T09:18:11.857 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2021-07-10T09:18:11.886 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:11.887 INFO:tasks.ceph:role: cluster1.osd.0 2021-07-10T09:18:11.887 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:18:11.888 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-07-10T09:18:11.963 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-07-10T09:18:11.963 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2021-07-10T09:18:11.963 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-07-10T09:18:11.964 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 2021-07-10T09:18:11.964 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-07-10T09:18:11.964 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2021-07-10T09:18:11.964 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-07-10T09:18:11.964 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-07-10T09:18:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-07-10T09:18:11.965 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-07-10T09:18:12.325 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2021-07-10T09:18:12.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2021-07-10T09:18:12.327 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2021-07-10T09:18:12.464 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2021-07-10T09:18:12.499 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2021-07-10T09:18:12.567 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:12.568 INFO:tasks.ceph:role: cluster1.osd.1 2021-07-10T09:18:12.568 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:18:12.568 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-07-10T09:18:12.642 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-07-10T09:18:12.643 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2021-07-10T09:18:12.643 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-07-10T09:18:12.643 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 2021-07-10T09:18:12.643 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-07-10T09:18:12.644 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2021-07-10T09:18:12.644 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-07-10T09:18:12.644 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-07-10T09:18:12.644 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-07-10T09:18:12.644 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-07-10T09:18:12.995 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2021-07-10T09:18:12.997 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2021-07-10T09:18:12.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2021-07-10T09:18:13.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2021-07-10T09:18:13.119 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2021-07-10T09:18:13.188 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:13.189 INFO:tasks.ceph:role: cluster1.osd.2 2021-07-10T09:18:13.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:18:13.190 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-07-10T09:18:13.265 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-07-10T09:18:13.266 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2021-07-10T09:18:13.266 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-07-10T09:18:13.266 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 2021-07-10T09:18:13.266 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-07-10T09:18:13.267 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2021-07-10T09:18:13.267 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-07-10T09:18:13.267 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-07-10T09:18:13.267 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-07-10T09:18:13.267 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-07-10T09:18:13.632 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2021-07-10T09:18:13.634 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2021-07-10T09:18:13.635 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2021-07-10T09:18:13.748 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2021-07-10T09:18:13.781 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:13.880 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:13.877+0000 7f4892f2d0c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2021-07-10T09:18:13.880 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:13.878+0000 7f4892f2d0c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2021-07-10T09:18:13.881 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:13.878+0000 7f4892f2d0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2021-07-10T09:18:13.881 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:13.878+0000 7f4892f2d0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2021-07-10T09:18:13.881 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:13.878+0000 7f4892f2d0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2021-07-10T09:18:13.881 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:13.878+0000 7f4892f2d0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2021-07-10T09:18:16.734 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2021-07-10T09:18:16.766 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:16.851+0000 7f303cf780c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2021-07-10T09:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:16.851+0000 7f303cf780c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2021-07-10T09:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:16.851+0000 7f303cf780c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2021-07-10T09:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:16.851+0000 7f303cf780c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2021-07-10T09:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:16.851+0000 7f303cf780c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2021-07-10T09:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:16.852+0000 7f303cf780c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2021-07-10T09:18:19.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2021-07-10T09:18:19.741 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:19.827+0000 7fc673b1e0c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2021-07-10T09:18:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:19.827+0000 7fc673b1e0c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2021-07-10T09:18:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:19.827+0000 7fc673b1e0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2021-07-10T09:18:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:19.827+0000 7fc673b1e0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2021-07-10T09:18:19.831 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:19.827+0000 7fc673b1e0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2021-07-10T09:18:19.831 INFO:teuthology.orchestra.run.smithi062.stderr:2021-07-10T09:18:19.828+0000 7fc673b1e0c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2021-07-10T09:18:22.682 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2021-07-10T09:18:22.714 INFO:tasks.ceph:Reading keys from all nodes... 2021-07-10T09:18:22.714 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:22.715 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2021-07-10T09:18:22.784 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:22.784 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2021-07-10T09:18:22.852 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:22.853 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2021-07-10T09:18:22.920 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:22.921 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2021-07-10T09:18:22.988 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:22.989 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2021-07-10T09:18:23.046 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.047 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2021-07-10T09:18:23.071 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.072 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2021-07-10T09:18:23.129 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.130 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2021-07-10T09:18:23.187 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.187 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2021-07-10T09:18:23.245 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.246 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2021-07-10T09:18:23.303 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.303 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2021-07-10T09:18:23.361 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.362 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2021-07-10T09:18:23.418 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:23.418 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2021-07-10T09:18:23.479 INFO:tasks.ceph:Adding keys to all mons... 2021-07-10T09:18:23.480 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/cluster1.keyring 2021-07-10T09:18:23.510 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2021-07-10T09:18:23.510 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgK9F9BhAAuvjR8uyyJMWrtoFLHM7b1w== 2021-07-10T09:18:23.511 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2021-07-10T09:18:23.511 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDVZelgD4tnNBAAdlsSZTE5Xf5MgEe4fFQqjA== 2021-07-10T09:18:23.511 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2021-07-10T09:18:23.511 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDYZelgnC3UMhAAKZ9Zg4U8EVQJtSmXN1ojdQ== 2021-07-10T09:18:23.511 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2021-07-10T09:18:23.512 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDbZelgq+BlMRAAxrU2nBdPnQli+OteVPSljg== 2021-07-10T09:18:23.512 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2021-07-10T09:18:23.512 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgEfMOCRAAixcbgbXq1lhejPw80yfupw== 2021-07-10T09:18:23.512 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror] 2021-07-10T09:18:23.513 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgKjkRDhAAX/5VVszOqelKEmHa61PJyA== 2021-07-10T09:18:23.513 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.0] 2021-07-10T09:18:23.513 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgQaRJERAA1X65S/VKLWcUdSDyYak4Og== 2021-07-10T09:18:23.513 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.1] 2021-07-10T09:18:23.513 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgcnn5FhAA9VLyeT1p2M9pPyFIydKOCw== 2021-07-10T09:18:23.514 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.2] 2021-07-10T09:18:23.514 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelg/ItXHBAAm2M8jnXvyBDaIkRhNZ6pjw== 2021-07-10T09:18:23.514 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.3] 2021-07-10T09:18:23.514 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgWF/HIRAAjZOoBEgDqVdD+9KrvDWQMg== 2021-07-10T09:18:23.515 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.4] 2021-07-10T09:18:23.515 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelg0VAfJxAAOSmYmzJFPdoUi4S3fQD+Ag== 2021-07-10T09:18:23.515 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.5] 2021-07-10T09:18:23.515 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgecmBLBAAJtZCqVF30MZfwWnstL6VYg== 2021-07-10T09:18:23.515 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.6] 2021-07-10T09:18:23.516 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTZelgkF3uMRAANPAs/lsEluCFV7bnfBPcUA== 2021-07-10T09:18:23.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-07-10T09:18:23.612 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-07-10T09:18:23.664 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-07-10T09:18:23.715 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-07-10T09:18:23.763 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:23.816 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:23.867 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:23.919 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:23.971 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:24.023 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:24.075 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:24.127 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:24.179 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:18:24.232 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-07-10T09:18:24.232 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2021-07-10T09:18:24.259 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2021-07-10T09:18:24.355 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2021-07-10T09:18:24.381 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2021-07-10T09:18:24.438 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2021-07-10T09:18:24.439 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2021-07-10T09:18:24.440 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2021-07-10T09:18:24.482 INFO:tasks.ceph.cluster1.mon.a:Started 2021-07-10T09:18:24.483 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2021-07-10T09:18:24.483 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2021-07-10T09:18:24.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2021-07-10T09:18:24.486 INFO:tasks.ceph.cluster1.mgr.x:Started 2021-07-10T09:18:24.487 INFO:tasks.ceph:Setting crush tunables to default 2021-07-10T09:18:24.487 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd crush tunables default 2021-07-10T09:18:24.579 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: vendor '' 2021-07-10T09:18:24.579 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:24.579 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:24.580 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:24.580 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:24.582 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: vendor '' 2021-07-10T09:18:24.583 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:24.583 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:24.583 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:24.583 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:24.695 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2021-07-10T09:18:24.706 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2021-07-10T09:18:24.708 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:24.709 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2021-07-10T09:18:24.735 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:24.736 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2021-07-10T09:18:24.801 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:24.802 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2021-07-10T09:18:24.869 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd new 6c410565-6eb7-4573-b034-75f005d1d633 0 2021-07-10T09:18:25.119 INFO:teuthology.orchestra.run.smithi062.stdout:0 2021-07-10T09:18:25.132 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd new 93616833-32c7-48f0-8135-c49115b03413 1 2021-07-10T09:18:25.351 INFO:teuthology.orchestra.run.smithi062.stdout:1 2021-07-10T09:18:25.364 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd new 5de23214-ea4e-4a03-a17f-912dbab980b1 2 2021-07-10T09:18:25.621 INFO:teuthology.orchestra.run.smithi062.stdout:2 2021-07-10T09:18:25.634 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2021-07-10T09:18:25.635 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2021-07-10T09:18:25.637 INFO:tasks.ceph.cluster1.osd.0:Started 2021-07-10T09:18:25.637 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2021-07-10T09:18:25.638 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2021-07-10T09:18:25.640 INFO:tasks.ceph.cluster1.osd.1:Started 2021-07-10T09:18:25.640 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2021-07-10T09:18:25.640 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2021-07-10T09:18:25.642 INFO:tasks.ceph.cluster1.osd.2:Started 2021-07-10T09:18:25.643 INFO:tasks.ceph:Waiting for OSDs to come up 2021-07-10T09:18:25.947 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2021-07-10T09:18:26.132 INFO:teuthology.misc.health.smithi062.stdout: 2021-07-10T09:18:26.133 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":5,"fsid":"7e08b311-22b7-4a7a-a310-0bd6eae2431d","created":"2021-07-10T09:18:24.583086+0000","modified":"2021-07-10T09:18:25.619217+0000","last_up_change":"0.000000","last_in_change":"2021-07-10T09:18:25.619217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6c410565-6eb7-4573-b034-75f005d1d633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"93616833-32c7-48f0-8135-c49115b03413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5de23214-ea4e-4a03-a17f-912dbab980b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:18:26.144 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-07-10T09:18:26.231 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:18:26.229+0000 7f72f21b90c0 -1 Falling back to public interface 2021-07-10T09:18:26.238 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:18:26.236+0000 7f31a018a0c0 -1 Falling back to public interface 2021-07-10T09:18:26.242 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:18:26.240+0000 7f4250a990c0 -1 Falling back to public interface 2021-07-10T09:18:29.723 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:18:29.721+0000 7f31a018a0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-10T09:18:29.731 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:18:29.729+0000 7f72f21b90c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-10T09:18:29.775 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:18:29.773+0000 7f4250a990c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-10T09:18:32.448 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2021-07-10T09:18:32.907 INFO:teuthology.misc.health.smithi062.stdout: 2021-07-10T09:18:32.908 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":7,"fsid":"7e08b311-22b7-4a7a-a310-0bd6eae2431d","created":"2021-07-10T09:18:24.583086+0000","modified":"2021-07-10T09:18:31.587721+0000","last_up_change":"0.000000","last_in_change":"2021-07-10T09:18:25.619217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6c410565-6eb7-4573-b034-75f005d1d633","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"93616833-32c7-48f0-8135-c49115b03413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5de23214-ea4e-4a03-a17f-912dbab980b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:18:32.930 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-07-10T09:18:37.061 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:18:37.059+0000 7f4247212700 -1 osd.2 0 waiting for initial osdmap 2021-07-10T09:18:37.070 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:18:37.068+0000 7f423fbb3700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-07-10T09:18:37.082 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: vendor '' 2021-07-10T09:18:37.082 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:37.082 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:37.083 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:37.083 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:37.103 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:18:37.101+0000 7f3196903700 -1 osd.0 0 waiting for initial osdmap 2021-07-10T09:18:37.110 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:18:37.108+0000 7f318f2a4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-07-10T09:18:37.111 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: vendor '' 2021-07-10T09:18:37.112 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:37.112 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:37.112 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:37.112 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:37.338 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:18:37.336+0000 7f72e8932700 -1 osd.1 0 waiting for initial osdmap 2021-07-10T09:18:37.343 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:18:37.341+0000 7f72e12d3700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-07-10T09:18:37.344 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: vendor '' 2021-07-10T09:18:37.345 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:37.345 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:37.345 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:37.345 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:39.234 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2021-07-10T09:18:39.535 INFO:teuthology.misc.health.smithi062.stdout: 2021-07-10T09:18:39.536 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":9,"fsid":"7e08b311-22b7-4a7a-a310-0bd6eae2431d","created":"2021-07-10T09:18:24.583086+0000","modified":"2021-07-10T09:18:38.613088+0000","last_up_change":"2021-07-10T09:18:37.608229+0000","last_in_change":"2021-07-10T09:18:25.619217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6c410565-6eb7-4573-b034-75f005d1d633","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":31730},{"type":"v1","addr":"172.21.15.62:6809","nonce":31730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":31730},{"type":"v1","addr":"172.21.15.62:6811","nonce":31730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":31730},{"type":"v1","addr":"172.21.15.62:6815","nonce":31730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":31730},{"type":"v1","addr":"172.21.15.62:6813","nonce":31730}]},"public_addr":"172.21.15.62:6809/31730","cluster_addr":"172.21.15.62:6811/31730","heartbeat_back_addr":"172.21.15.62:6815/31730","heartbeat_front_addr":"172.21.15.62:6813/31730","state":["exists","up"]},{"osd":1,"uuid":"93616833-32c7-48f0-8135-c49115b03413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":31731},{"type":"v1","addr":"172.21.15.62:6801","nonce":31731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":31731},{"type":"v1","addr":"172.21.15.62:6803","nonce":31731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":31731},{"type":"v1","addr":"172.21.15.62:6807","nonce":31731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":31731},{"type":"v1","addr":"172.21.15.62:6805","nonce":31731}]},"public_addr":"172.21.15.62:6801/31731","cluster_addr":"172.21.15.62:6803/31731","heartbeat_back_addr":"172.21.15.62:6807/31731","heartbeat_front_addr":"172.21.15.62:6805/31731","state":["exists","up"]},{"osd":2,"uuid":"5de23214-ea4e-4a03-a17f-912dbab980b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":31732},{"type":"v1","addr":"172.21.15.62:6817","nonce":31732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":31732},{"type":"v1","addr":"172.21.15.62:6819","nonce":31732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":31732},{"type":"v1","addr":"172.21.15.62:6823","nonce":31732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":31732},{"type":"v1","addr":"172.21.15.62:6821","nonce":31732}]},"public_addr":"172.21.15.62:6817/31732","cluster_addr":"172.21.15.62:6819/31732","heartbeat_back_addr":"172.21.15.62:6823/31732","heartbeat_front_addr":"172.21.15.62:6821/31732","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.680895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.770408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.735442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:18:39.548 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-07-10T09:18:39.549 INFO:tasks.ceph:Creating RBD pool 2021-07-10T09:18:39.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd pool create rbd 8 2021-07-10T09:18:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'rbd' created 2021-07-10T09:18:40.645 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-07-10T09:18:40.750 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: vendor '' 2021-07-10T09:18:40.750 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:40.750 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:40.750 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:40.751 INFO:tasks.ceph.cluster1.osd.0.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:40.903 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: vendor '' 2021-07-10T09:18:40.903 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:40.903 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:40.903 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:40.904 INFO:tasks.ceph.cluster1.osd.1.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:41.000 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: vendor '' 2021-07-10T09:18:41.000 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:41.000 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:41.001 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:41.001 INFO:tasks.ceph.cluster1.osd.2.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:41.097 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: vendor '' 2021-07-10T09:18:41.097 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:18:41.097 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: serial_short 'CVFT53440094400BGN' 2021-07-10T09:18:41.097 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: scsi_serial '' 2021-07-10T09:18:41.098 INFO:tasks.ceph.cluster1.mon.a.smithi062.stdout: serial 'INTEL SSDPEDMD400G4_CVFT53440094400BGN' 2021-07-10T09:18:41.627 INFO:teuthology.orchestra.run.smithi062.stderr:enabled application 'rbd' on pool 'rbd' 2021-07-10T09:18:41.642 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2021-07-10T09:18:41.643 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2021-07-10T09:18:41.644 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-07-10T09:18:41.940 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:41.940 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":12,"fsid":"7e08b311-22b7-4a7a-a310-0bd6eae2431d","created":"2021-07-10T09:18:24.583086+0000","modified":"2021-07-10T09:18:41.618772+0000","last_up_change":"2021-07-10T09:18:37.608229+0000","last_in_change":"2021-07-10T09:18:25.619217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-07-10T09:18:39.593274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-10T09:18:39.891621+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"6c410565-6eb7-4573-b034-75f005d1d633","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":31730},{"type":"v1","addr":"172.21.15.62:6809","nonce":31730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":31730},{"type":"v1","addr":"172.21.15.62:6811","nonce":31730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":31730},{"type":"v1","addr":"172.21.15.62:6815","nonce":31730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":31730},{"type":"v1","addr":"172.21.15.62:6813","nonce":31730}]},"public_addr":"172.21.15.62:6809/31730","cluster_addr":"172.21.15.62:6811/31730","heartbeat_back_addr":"172.21.15.62:6815/31730","heartbeat_front_addr":"172.21.15.62:6813/31730","state":["exists","up"]},{"osd":1,"uuid":"93616833-32c7-48f0-8135-c49115b03413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":31731},{"type":"v1","addr":"172.21.15.62:6801","nonce":31731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":31731},{"type":"v1","addr":"172.21.15.62:6803","nonce":31731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":31731},{"type":"v1","addr":"172.21.15.62:6807","nonce":31731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":31731},{"type":"v1","addr":"172.21.15.62:6805","nonce":31731}]},"public_addr":"172.21.15.62:6801/31731","cluster_addr":"172.21.15.62:6803/31731","heartbeat_back_addr":"172.21.15.62:6807/31731","heartbeat_front_addr":"172.21.15.62:6805/31731","state":["exists","up"]},{"osd":2,"uuid":"5de23214-ea4e-4a03-a17f-912dbab980b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":31732},{"type":"v1","addr":"172.21.15.62:6817","nonce":31732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":31732},{"type":"v1","addr":"172.21.15.62:6819","nonce":31732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":31732},{"type":"v1","addr":"172.21.15.62:6823","nonce":31732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":31732},{"type":"v1","addr":"172.21.15.62:6821","nonce":31732}]},"public_addr":"172.21.15.62:6817/31732","cluster_addr":"172.21.15.62:6819/31732","heartbeat_back_addr":"172.21.15.62:6823/31732","heartbeat_front_addr":"172.21.15.62:6821/31732","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.680895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.770408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.735442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:18:41.955 INFO:tasks.ceph.ceph_manager.cluster1:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2021-07-10T09:18:39.593274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-07-10T09:18:39.891621+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-07-10T09:18:41.958 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd pool get .mgr pg_num 2021-07-10T09:18:42.252 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2021-07-10T09:18:42.265 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd pool get rbd pg_num 2021-07-10T09:18:42.561 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 8 2021-07-10T09:18:42.574 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2021-07-10T09:18:42.575 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2021-07-10T09:18:42.575 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2021-07-10T09:18:42.915 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:42.928 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":31386},{"type":"v1","addr":"172.21.15.62:6825","nonce":31386}]},"active_addr":"172.21.15.62:6825/31386","active_change":"2021-07-10T09:18:29.528032+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2237335455}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":4223528216}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":3722135217}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2584915083}]}]}} 2021-07-10T09:18:42.931 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2021-07-10T09:18:42.931 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2021-07-10T09:18:42.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2021-07-10T09:18:43.227 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:43.227 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":13,"fsid":"7e08b311-22b7-4a7a-a310-0bd6eae2431d","created":"2021-07-10T09:18:24.583086+0000","modified":"2021-07-10T09:18:42.627159+0000","last_up_change":"2021-07-10T09:18:37.608229+0000","last_in_change":"2021-07-10T09:18:25.619217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-07-10T09:18:39.593274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-10T09:18:39.891621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"6c410565-6eb7-4573-b034-75f005d1d633","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":31730},{"type":"v1","addr":"172.21.15.62:6809","nonce":31730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":31730},{"type":"v1","addr":"172.21.15.62:6811","nonce":31730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":31730},{"type":"v1","addr":"172.21.15.62:6815","nonce":31730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":31730},{"type":"v1","addr":"172.21.15.62:6813","nonce":31730}]},"public_addr":"172.21.15.62:6809/31730","cluster_addr":"172.21.15.62:6811/31730","heartbeat_back_addr":"172.21.15.62:6815/31730","heartbeat_front_addr":"172.21.15.62:6813/31730","state":["exists","up"]},{"osd":1,"uuid":"93616833-32c7-48f0-8135-c49115b03413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":31731},{"type":"v1","addr":"172.21.15.62:6801","nonce":31731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":31731},{"type":"v1","addr":"172.21.15.62:6803","nonce":31731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":31731},{"type":"v1","addr":"172.21.15.62:6807","nonce":31731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":31731},{"type":"v1","addr":"172.21.15.62:6805","nonce":31731}]},"public_addr":"172.21.15.62:6801/31731","cluster_addr":"172.21.15.62:6803/31731","heartbeat_back_addr":"172.21.15.62:6807/31731","heartbeat_front_addr":"172.21.15.62:6805/31731","state":["exists","up"]},{"osd":2,"uuid":"5de23214-ea4e-4a03-a17f-912dbab980b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":31732},{"type":"v1","addr":"172.21.15.62:6817","nonce":31732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":31732},{"type":"v1","addr":"172.21.15.62:6819","nonce":31732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":31732},{"type":"v1","addr":"172.21.15.62:6823","nonce":31732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":31732},{"type":"v1","addr":"172.21.15.62:6821","nonce":31732}]},"public_addr":"172.21.15.62:6817/31732","cluster_addr":"172.21.15.62:6819/31732","heartbeat_back_addr":"172.21.15.62:6823/31732","heartbeat_front_addr":"172.21.15.62:6821/31732","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.680895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.770408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.735442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:18:43.240 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2021-07-10T09:18:43.241 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2021-07-10T09:18:43.537 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:43.537 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":13,"fsid":"7e08b311-22b7-4a7a-a310-0bd6eae2431d","created":"2021-07-10T09:18:24.583086+0000","modified":"2021-07-10T09:18:42.627159+0000","last_up_change":"2021-07-10T09:18:37.608229+0000","last_in_change":"2021-07-10T09:18:25.619217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-07-10T09:18:39.593274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-10T09:18:39.891621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"6c410565-6eb7-4573-b034-75f005d1d633","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":31730},{"type":"v1","addr":"172.21.15.62:6809","nonce":31730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":31730},{"type":"v1","addr":"172.21.15.62:6811","nonce":31730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":31730},{"type":"v1","addr":"172.21.15.62:6815","nonce":31730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":31730},{"type":"v1","addr":"172.21.15.62:6813","nonce":31730}]},"public_addr":"172.21.15.62:6809/31730","cluster_addr":"172.21.15.62:6811/31730","heartbeat_back_addr":"172.21.15.62:6815/31730","heartbeat_front_addr":"172.21.15.62:6813/31730","state":["exists","up"]},{"osd":1,"uuid":"93616833-32c7-48f0-8135-c49115b03413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":31731},{"type":"v1","addr":"172.21.15.62:6801","nonce":31731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":31731},{"type":"v1","addr":"172.21.15.62:6803","nonce":31731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":31731},{"type":"v1","addr":"172.21.15.62:6807","nonce":31731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":31731},{"type":"v1","addr":"172.21.15.62:6805","nonce":31731}]},"public_addr":"172.21.15.62:6801/31731","cluster_addr":"172.21.15.62:6803/31731","heartbeat_back_addr":"172.21.15.62:6807/31731","heartbeat_front_addr":"172.21.15.62:6805/31731","state":["exists","up"]},{"osd":2,"uuid":"5de23214-ea4e-4a03-a17f-912dbab980b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":31732},{"type":"v1","addr":"172.21.15.62:6817","nonce":31732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":31732},{"type":"v1","addr":"172.21.15.62:6819","nonce":31732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":31732},{"type":"v1","addr":"172.21.15.62:6823","nonce":31732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":31732},{"type":"v1","addr":"172.21.15.62:6821","nonce":31732}]},"public_addr":"172.21.15.62:6817/31732","cluster_addr":"172.21.15.62:6819/31732","heartbeat_back_addr":"172.21.15.62:6823/31732","heartbeat_front_addr":"172.21.15.62:6821/31732","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.680895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.770408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:18:30.735442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:18:43.551 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2021-07-10T09:18:43.704 INFO:teuthology.orchestra.run.smithi062.stdout:34359738374 2021-07-10T09:18:43.705 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2021-07-10T09:18:43.858 INFO:teuthology.orchestra.run.smithi062.stdout:34359738377 2021-07-10T09:18:43.858 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2021-07-10T09:18:44.012 INFO:teuthology.orchestra.run.smithi062.stdout:34359738375 2021-07-10T09:18:44.014 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2021-07-10T09:18:44.309 INFO:teuthology.orchestra.run.smithi062.stdout:34359738372 2021-07-10T09:18:44.323 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.0 2021-07-10T09:18:45.325 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2021-07-10T09:18:45.634 INFO:teuthology.orchestra.run.smithi062.stdout:34359738372 2021-07-10T09:18:45.649 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.0 2021-07-10T09:18:46.650 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2021-07-10T09:18:46.959 INFO:teuthology.orchestra.run.smithi062.stdout:34359738374 2021-07-10T09:18:46.973 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.0 2021-07-10T09:18:46.974 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2021-07-10T09:18:47.269 INFO:teuthology.orchestra.run.smithi062.stdout:34359738377 2021-07-10T09:18:47.282 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738377 got 34359738377 for osd.1 2021-07-10T09:18:47.282 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2021-07-10T09:18:47.577 INFO:teuthology.orchestra.run.smithi062.stdout:34359738375 2021-07-10T09:18:47.590 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738375 for osd.2 2021-07-10T09:18:47.591 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2021-07-10T09:18:47.592 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2021-07-10T09:18:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:47.885 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2021-07-10T09:18:47.896 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-07-10T09:18:47.533796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":52544,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":51392,"kb_avail":283062976,"statfs":{"total":289910292480,"available":289856487424,"internally_reserved":0,"allocated":958464,"data_stored":1269200,"data_compressed":60064,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":52625408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000315"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634622+0000","last_change":"2021-07-10T09:18:41.634622+0000","last_active":"2021-07-10T09:18:41.634622+0000","last_peered":"2021-07-10T09:18:41.634622+0000","last_clean":"2021-07-10T09:18:41.634622+0000","last_became_active":"2021-07-10T09:18:41.634357+0000","last_became_peered":"2021-07-10T09:18:41.634357+0000","last_unstale":"2021-07-10T09:18:41.634622+0000","last_undegraded":"2021-07-10T09:18:41.634622+0000","last_fullsized":"2021-07-10T09:18:41.634622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634570+0000","last_change":"2021-07-10T09:18:41.634570+0000","last_active":"2021-07-10T09:18:41.634570+0000","last_peered":"2021-07-10T09:18:41.634570+0000","last_clean":"2021-07-10T09:18:41.634570+0000","last_became_active":"2021-07-10T09:18:41.634294+0000","last_became_peered":"2021-07-10T09:18:41.634294+0000","last_unstale":"2021-07-10T09:18:41.634570+0000","last_undegraded":"2021-07-10T09:18:41.634570+0000","last_fullsized":"2021-07-10T09:18:41.634570+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.635186+0000","last_change":"2021-07-10T09:18:41.635186+0000","last_active":"2021-07-10T09:18:41.635186+0000","last_peered":"2021-07-10T09:18:41.635186+0000","last_clean":"2021-07-10T09:18:41.635186+0000","last_became_active":"2021-07-10T09:18:41.634856+0000","last_became_peered":"2021-07-10T09:18:41.634856+0000","last_unstale":"2021-07-10T09:18:41.635186+0000","last_undegraded":"2021-07-10T09:18:41.635186+0000","last_fullsized":"2021-07-10T09:18:41.635186+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634166+0000","last_change":"2021-07-10T09:18:41.632802+0000","last_active":"2021-07-10T09:18:41.634166+0000","last_peered":"2021-07-10T09:18:41.634166+0000","last_clean":"2021-07-10T09:18:41.634166+0000","last_became_active":"2021-07-10T09:18:41.632375+0000","last_became_peered":"2021-07-10T09:18:41.632375+0000","last_unstale":"2021-07-10T09:18:41.634166+0000","last_undegraded":"2021-07-10T09:18:41.634166+0000","last_fullsized":"2021-07-10T09:18:41.634166+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.205400+0000","last_change":"2021-07-10T09:18:40.628678+0000","last_active":"2021-07-10T09:18:41.205400+0000","last_peered":"2021-07-10T09:18:41.205400+0000","last_clean":"2021-07-10T09:18:41.205400+0000","last_became_active":"2021-07-10T09:18:40.627651+0000","last_became_peered":"2021-07-10T09:18:40.627651+0000","last_unstale":"2021-07-10T09:18:41.205400+0000","last_undegraded":"2021-07-10T09:18:41.205400+0000","last_fullsized":"2021-07-10T09:18:41.205400+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:39.616058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:39.616058+0000","last_clean_scrub_stamp":"2021-07-10T09:18:39.616058+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.632713+0000","last_change":"2021-07-10T09:18:41.632713+0000","last_active":"2021-07-10T09:18:41.632713+0000","last_peered":"2021-07-10T09:18:41.632713+0000","last_clean":"2021-07-10T09:18:41.632713+0000","last_became_active":"2021-07-10T09:18:41.632283+0000","last_became_peered":"2021-07-10T09:18:41.632283+0000","last_unstale":"2021-07-10T09:18:41.632713+0000","last_undegraded":"2021-07-10T09:18:41.632713+0000","last_fullsized":"2021-07-10T09:18:41.632713+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634239+0000","last_change":"2021-07-10T09:18:41.633027+0000","last_active":"2021-07-10T09:18:41.634239+0000","last_peered":"2021-07-10T09:18:41.634239+0000","last_clean":"2021-07-10T09:18:41.634239+0000","last_became_active":"2021-07-10T09:18:41.632289+0000","last_became_peered":"2021-07-10T09:18:41.632289+0000","last_unstale":"2021-07-10T09:18:41.634239+0000","last_undegraded":"2021-07-10T09:18:41.634239+0000","last_fullsized":"2021-07-10T09:18:41.634239+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.636048+0000","last_change":"2021-07-10T09:18:41.636048+0000","last_active":"2021-07-10T09:18:41.636048+0000","last_peered":"2021-07-10T09:18:41.636048+0000","last_clean":"2021-07-10T09:18:41.636048+0000","last_became_active":"2021-07-10T09:18:41.635760+0000","last_became_peered":"2021-07-10T09:18:41.635760+0000","last_unstale":"2021-07-10T09:18:41.636048+0000","last_undegraded":"2021-07-10T09:18:41.636048+0000","last_fullsized":"2021-07-10T09:18:41.636048+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634498+0000","last_change":"2021-07-10T09:18:41.634498+0000","last_active":"2021-07-10T09:18:41.634498+0000","last_peered":"2021-07-10T09:18:41.634498+0000","last_clean":"2021-07-10T09:18:41.634498+0000","last_became_active":"2021-07-10T09:18:41.634253+0000","last_became_peered":"2021-07-10T09:18:41.634253+0000","last_unstale":"2021-07-10T09:18:41.634498+0000","last_undegraded":"2021-07-10T09:18:41.634498+0000","last_fullsized":"2021-07-10T09:18:41.634498+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":1180736,"data_compressed":60064,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16952,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":16832,"kb_avail":94354888,"statfs":{"total":96636764160,"available":96619405312,"internally_reserved":0,"allocated":114688,"data_stored":31810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17235968},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17796,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354044,"statfs":{"total":96636764160,"available":96618541056,"internally_reserved":0,"allocated":421888,"data_stored":618695,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17796,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354044,"statfs":{"total":96636764160,"available":96618541056,"internally_reserved":0,"allocated":421888,"data_stored":618695,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-07-10T09:18:47.898 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2021-07-10T09:18:48.186 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:48.188 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2021-07-10T09:18:48.199 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-07-10T09:18:47.533796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":52544,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":51392,"kb_avail":283062976,"statfs":{"total":289910292480,"available":289856487424,"internally_reserved":0,"allocated":958464,"data_stored":1269200,"data_compressed":60064,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":52625408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000315"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634622+0000","last_change":"2021-07-10T09:18:41.634622+0000","last_active":"2021-07-10T09:18:41.634622+0000","last_peered":"2021-07-10T09:18:41.634622+0000","last_clean":"2021-07-10T09:18:41.634622+0000","last_became_active":"2021-07-10T09:18:41.634357+0000","last_became_peered":"2021-07-10T09:18:41.634357+0000","last_unstale":"2021-07-10T09:18:41.634622+0000","last_undegraded":"2021-07-10T09:18:41.634622+0000","last_fullsized":"2021-07-10T09:18:41.634622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634570+0000","last_change":"2021-07-10T09:18:41.634570+0000","last_active":"2021-07-10T09:18:41.634570+0000","last_peered":"2021-07-10T09:18:41.634570+0000","last_clean":"2021-07-10T09:18:41.634570+0000","last_became_active":"2021-07-10T09:18:41.634294+0000","last_became_peered":"2021-07-10T09:18:41.634294+0000","last_unstale":"2021-07-10T09:18:41.634570+0000","last_undegraded":"2021-07-10T09:18:41.634570+0000","last_fullsized":"2021-07-10T09:18:41.634570+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.635186+0000","last_change":"2021-07-10T09:18:41.635186+0000","last_active":"2021-07-10T09:18:41.635186+0000","last_peered":"2021-07-10T09:18:41.635186+0000","last_clean":"2021-07-10T09:18:41.635186+0000","last_became_active":"2021-07-10T09:18:41.634856+0000","last_became_peered":"2021-07-10T09:18:41.634856+0000","last_unstale":"2021-07-10T09:18:41.635186+0000","last_undegraded":"2021-07-10T09:18:41.635186+0000","last_fullsized":"2021-07-10T09:18:41.635186+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634166+0000","last_change":"2021-07-10T09:18:41.632802+0000","last_active":"2021-07-10T09:18:41.634166+0000","last_peered":"2021-07-10T09:18:41.634166+0000","last_clean":"2021-07-10T09:18:41.634166+0000","last_became_active":"2021-07-10T09:18:41.632375+0000","last_became_peered":"2021-07-10T09:18:41.632375+0000","last_unstale":"2021-07-10T09:18:41.634166+0000","last_undegraded":"2021-07-10T09:18:41.634166+0000","last_fullsized":"2021-07-10T09:18:41.634166+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.205400+0000","last_change":"2021-07-10T09:18:40.628678+0000","last_active":"2021-07-10T09:18:41.205400+0000","last_peered":"2021-07-10T09:18:41.205400+0000","last_clean":"2021-07-10T09:18:41.205400+0000","last_became_active":"2021-07-10T09:18:40.627651+0000","last_became_peered":"2021-07-10T09:18:40.627651+0000","last_unstale":"2021-07-10T09:18:41.205400+0000","last_undegraded":"2021-07-10T09:18:41.205400+0000","last_fullsized":"2021-07-10T09:18:41.205400+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:39.616058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:39.616058+0000","last_clean_scrub_stamp":"2021-07-10T09:18:39.616058+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.632713+0000","last_change":"2021-07-10T09:18:41.632713+0000","last_active":"2021-07-10T09:18:41.632713+0000","last_peered":"2021-07-10T09:18:41.632713+0000","last_clean":"2021-07-10T09:18:41.632713+0000","last_became_active":"2021-07-10T09:18:41.632283+0000","last_became_peered":"2021-07-10T09:18:41.632283+0000","last_unstale":"2021-07-10T09:18:41.632713+0000","last_undegraded":"2021-07-10T09:18:41.632713+0000","last_fullsized":"2021-07-10T09:18:41.632713+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634239+0000","last_change":"2021-07-10T09:18:41.633027+0000","last_active":"2021-07-10T09:18:41.634239+0000","last_peered":"2021-07-10T09:18:41.634239+0000","last_clean":"2021-07-10T09:18:41.634239+0000","last_became_active":"2021-07-10T09:18:41.632289+0000","last_became_peered":"2021-07-10T09:18:41.632289+0000","last_unstale":"2021-07-10T09:18:41.634239+0000","last_undegraded":"2021-07-10T09:18:41.634239+0000","last_fullsized":"2021-07-10T09:18:41.634239+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.636048+0000","last_change":"2021-07-10T09:18:41.636048+0000","last_active":"2021-07-10T09:18:41.636048+0000","last_peered":"2021-07-10T09:18:41.636048+0000","last_clean":"2021-07-10T09:18:41.636048+0000","last_became_active":"2021-07-10T09:18:41.635760+0000","last_became_peered":"2021-07-10T09:18:41.635760+0000","last_unstale":"2021-07-10T09:18:41.636048+0000","last_undegraded":"2021-07-10T09:18:41.636048+0000","last_fullsized":"2021-07-10T09:18:41.636048+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:18:41.634498+0000","last_change":"2021-07-10T09:18:41.634498+0000","last_active":"2021-07-10T09:18:41.634498+0000","last_peered":"2021-07-10T09:18:41.634498+0000","last_clean":"2021-07-10T09:18:41.634498+0000","last_became_active":"2021-07-10T09:18:41.634253+0000","last_became_peered":"2021-07-10T09:18:41.634253+0000","last_unstale":"2021-07-10T09:18:41.634498+0000","last_undegraded":"2021-07-10T09:18:41.634498+0000","last_fullsized":"2021-07-10T09:18:41.634498+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:18:40.617867+0000","last_clean_scrub_stamp":"2021-07-10T09:18:40.617867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":1180736,"data_compressed":60064,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16952,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":16832,"kb_avail":94354888,"statfs":{"total":96636764160,"available":96619405312,"internally_reserved":0,"allocated":114688,"data_stored":31810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17235968},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17796,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354044,"statfs":{"total":96636764160,"available":96618541056,"internally_reserved":0,"allocated":421888,"data_stored":618695,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17796,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354044,"statfs":{"total":96636764160,"available":96618541056,"internally_reserved":0,"allocated":421888,"data_stored":618695,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30032,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-07-10T09:18:48.200 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2021-07-10T09:18:48.201 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2021-07-10T09:18:48.201 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2021-07-10T09:18:48.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2021-07-10T09:18:48.556 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:48.556 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-07-10T09:18:48.569 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2021-07-10T09:18:48.570 INFO:teuthology.run_tasks:Running task ceph... 2021-07-10T09:18:48.580 INFO:tasks.ceph:Creating ceph cluster cluster2... 2021-07-10T09:18:48.581 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-07-10T09:18:48.581 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi/6262049', 'branch': 'wip-mgolub-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'to.my.trociny@gmail.com', 'first_in_suite': False, 'job_id': '6262049', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'trociny-2021-07-10_06:51:06-rbd-wip-mgolub-testing-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-mgolub-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mgolub-testing', 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, 'owner': 'scheduled_trociny@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b', 'sleep_before_teardown': 0, 'suite': 'rbd', 'suite_branch': 'wip-mgolub-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '92adad3411b7280fb159eb9d7304c654695beb8b', 'targets': {'smithi062.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjifRisnkBt6E3gXwjqJegeGV3N9KQSLRR3AhOQa1kN790cPaqd8ZBjIE8kHyKSQyoidtLxSE6vJTf07GZRht4joRBX8QVlr98pCqRD+h4zIAlbByupFYseek67lfsGdjW4v8Wrg1ETeERn89rLLhS2njcgQS/ZIRlchOeOuUhjrE+yGbYr7uI0VXp1zbVUsd0OjCyzDJUtOp8bDehlbGPSYS5jmYzfsaH3vDUlWP9xV1GmvyAGJPOXvaaXD2spe9luaLYS55UoQF60M+eSKciQPP0rCNnJqKnpnuRoiuw2hvgrxIwE/TSIbQnBoOqNIKL/dzhhul0TiGitVxelXTZwbdCri2ioPElcJ4YfOru9Lq2+oz9Vpvhr+IuABmzym+5atHmyOII5ueQ9WGTHUoB81A4jfpUBF1DV1vfelvLayrTaFbFyAo7zJwqs5KPy8+/q2egbJtXmnvBF1a5nF/iF+tDfwnGKVSD5/pK9WU79Vi5pmc04F4Gc80yf1haF+0=', 'smithi105.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpGytXlOFlN1ptltJGikwEmNTcon0D67LpimkU8fksWbJap5NMtH0HZq37iy6W2NttOaC/S3NQ2iZ1lyYqX052aPHMgH2IIs70+sUELig4BxOJj6fri+23DBkfXaM+nCQcFlrMvOmJyJiRWaG9MNFXAPOHIMJrBu3CadnSvkkfia/W7f2FxwYqb3Ky7xKf1DUw2lL6UVDYG90t0/aYIPvykDntRlFPBWO1SCQ8hY/ZZzR7ZiXShJ2xT0NiRYrMw1GOJ5cCWKTurL29gr/At8h27Y4hHY7W+3Y9JKap5O1a3rMAcwdrQ7+JEI6QwyxZxE6pzWy8yLbLCp1pdp+9MkF07TJTUCImxVWh/QpYOMYqcxaxCxS/pVorKn3kUi8QPobCLFMRpQGBZTMAXPdDD/ScmifnWnwFWiNedppxdMiF8oJN30gAhqzHg21kki+m2YP4R1fgKleUVKD+3LrYvOy5preWF6hj14c8/80znrRb4O/gKKsU5mchA/6dT2qgv1M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '92adad3411b7280fb159eb9d7304c654695beb8b'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology_branch': 'master', 'teuthology_sha1': '24e3280d7d1fc156628af95b24b27a29e38b2766', 'timestamp': '2021-07-10_06:51:06', 'tube': 'smithi', 'user': 'trociny', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-07-10T09:18:48.582 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2021-07-10T09:18:48.587 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2021-07-10T09:18:48.606 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2021-07-10T09:18:48.643 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/run/ceph 2021-07-10T09:18:48.676 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:48.676 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2021-07-10T09:18:48.734 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-07-10T09:18:48.734 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-07-10T09:18:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-07-10T09:18:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:48.792 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 108884 Links: 1 2021-07-10T09:18:48.792 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:48.792 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:48.792 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2021-07-10 09:17:59.461192335 +0000 2021-07-10T09:18:48.793 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2021-07-10 09:15:06.835209639 +0000 2021-07-10T09:18:48.793 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2021-07-10 09:15:06.835209639 +0000 2021-07-10T09:18:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2021-07-10T09:18:48.802 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2021-07-10T09:18:48.802 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2021-07-10T09:18:48.803 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000261137 s, 2.0 MB/s 2021-07-10T09:18:48.806 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-07-10T09:18:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-07-10T09:18:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:48.864 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 106372 Links: 1 2021-07-10T09:18:48.864 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:48.864 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:48.864 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2021-07-10 09:17:59.461192335 +0000 2021-07-10T09:18:48.865 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2021-07-10 09:15:07.168202948 +0000 2021-07-10T09:18:48.865 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2021-07-10 09:15:07.168202948 +0000 2021-07-10T09:18:48.865 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2021-07-10T09:18:48.873 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2021-07-10T09:18:48.874 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2021-07-10T09:18:48.874 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000206349 s, 2.5 MB/s 2021-07-10T09:18:48.877 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-07-10T09:18:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-07-10T09:18:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:48.936 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 107921 Links: 1 2021-07-10T09:18:48.936 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:48.937 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:48.937 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2021-07-10 09:17:59.461192335 +0000 2021-07-10T09:18:48.937 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2021-07-10 09:15:07.502196235 +0000 2021-07-10T09:18:48.937 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2021-07-10 09:15:07.502196235 +0000 2021-07-10T09:18:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2021-07-10T09:18:48.946 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2021-07-10T09:18:48.946 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2021-07-10T09:18:48.946 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000193497 s, 2.6 MB/s 2021-07-10T09:18:48.948 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-07-10T09:18:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-07-10T09:18:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-07-10T09:18:49.006 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 111636 Links: 1 2021-07-10T09:18:49.006 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-07-10T09:18:49.006 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2021-07-10T09:18:49.006 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2021-07-10 09:17:59.461192335 +0000 2021-07-10T09:18:49.007 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2021-07-10 09:15:07.861189021 +0000 2021-07-10T09:18:49.007 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2021-07-10 09:15:07.861189021 +0000 2021-07-10T09:18:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2021-07-10T09:18:49.015 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2021-07-10T09:18:49.016 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2021-07-10T09:18:49.016 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000199512 s, 2.6 MB/s 2021-07-10T09:18:49.023 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:49.024 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-07-10T09:18:49.024 INFO:tasks.ceph:Generating config... 2021-07-10T09:18:49.025 INFO:tasks.ceph:[client] debug rbd = 20 2021-07-10T09:18:49.026 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2021-07-10T09:18:49.026 INFO:tasks.ceph:[client] log to stderr = False 2021-07-10T09:18:49.026 INFO:tasks.ceph:[client] rbd default features = 125 2021-07-10T09:18:49.026 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2021-07-10T09:18:49.027 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.027 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.027 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.028 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.028 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.028 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.029 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.029 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.029 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-07-10T09:18:49.029 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2021-07-10T09:18:49.030 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-07-10T09:18:49.030 INFO:tasks.ceph:[mgr] debug ms = 1 2021-07-10T09:18:49.030 INFO:tasks.ceph:[mon] debug mon = 20 2021-07-10T09:18:49.031 INFO:tasks.ceph:[mon] debug ms = 1 2021-07-10T09:18:49.031 INFO:tasks.ceph:[mon] debug paxos = 20 2021-07-10T09:18:49.031 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-07-10T09:18:49.031 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2021-07-10T09:18:49.032 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-07-10T09:18:49.032 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-07-10T09:18:49.032 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2021-07-10T09:18:49.033 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2021-07-10T09:18:49.033 INFO:tasks.ceph:[osd] debug ms = 1 2021-07-10T09:18:49.034 INFO:tasks.ceph:[osd] debug osd = 20 2021-07-10T09:18:49.034 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2021-07-10T09:18:49.034 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-07-10T09:18:49.035 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-07-10T09:18:49.035 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-07-10T09:18:49.035 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-07-10T09:18:49.035 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-07-10T09:18:49.036 INFO:tasks.ceph:Setting up cluster2.mon.a... 2021-07-10T09:18:49.036 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2021-07-10T09:18:49.100 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.keyring 2021-07-10T09:18:49.124 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2021-07-10T09:18:49.211 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2021-07-10T09:18:49.280 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.105')] 2021-07-10T09:18:49.281 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.105', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2021-07-10T09:18:49.282 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:49.282 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-07-10T09:18:49.336 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.105 --print /home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:18:49.413 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:18:49.414 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: generated fsid a9719d92-9687-4432-ad8f-7fcc7e386a67 2021-07-10T09:18:49.414 INFO:teuthology.orchestra.run.smithi105.stdout:setting min_mon_release = octopus 2021-07-10T09:18:49.414 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 0 2021-07-10T09:18:49.414 INFO:teuthology.orchestra.run.smithi105.stdout:fsid a9719d92-9687-4432-ad8f-7fcc7e386a67 2021-07-10T09:18:49.415 INFO:teuthology.orchestra.run.smithi105.stdout:last_changed 2021-07-10T09:18:49.412877+0000 2021-07-10T09:18:49.415 INFO:teuthology.orchestra.run.smithi105.stdout:created 2021-07-10T09:18:49.412877+0000 2021-07-10T09:18:49.415 INFO:teuthology.orchestra.run.smithi105.stdout:min_mon_release 15 (octopus) 2021-07-10T09:18:49.415 INFO:teuthology.orchestra.run.smithi105.stdout:election_strategy: 1 2021-07-10T09:18:49.416 INFO:teuthology.orchestra.run.smithi105.stdout:0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2021-07-10T09:18:49.416 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2021-07-10T09:18:49.417 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-07-10T09:18:49.473 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID a9719d92-9687-4432-ad8f-7fcc7e386a67... 2021-07-10T09:18:49.475 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2021-07-10T09:18:49.477 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2021-07-10T09:18:49.526 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2021-07-10T09:18:49.526 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2021-07-10T09:18:49.527 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-07-10T09:18:49.527 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2021-07-10T09:18:49.527 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.527 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2021-07-10T09:18:49.528 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.528 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2021-07-10T09:18:49.528 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.528 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2021-07-10T09:18:49.528 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2021-07-10T09:18:49.529 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.529 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2021-07-10T09:18:49.529 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2021-07-10T09:18:49.529 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.529 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2021-07-10T09:18:49.530 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2021-07-10T09:18:49.530 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.530 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2021-07-10T09:18:49.530 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2021-07-10T09:18:49.531 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.531 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2021-07-10T09:18:49.531 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.531 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2021-07-10T09:18:49.531 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2021-07-10T09:18:49.532 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2021-07-10T09:18:49.532 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2021-07-10T09:18:49.532 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2021-07-10T09:18:49.532 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2021-07-10T09:18:49.532 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2021-07-10T09:18:49.533 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-07-10T09:18:49.533 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2021-07-10T09:18:49.533 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2021-07-10T09:18:49.533 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.533 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-07-10T09:18:49.534 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.534 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2021-07-10T09:18:49.534 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.534 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2021-07-10T09:18:49.534 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.535 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2021-07-10T09:18:49.535 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2021-07-10T09:18:49.535 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2021-07-10T09:18:49.535 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = 172.21.15.105 2021-07-10T09:18:49.536 INFO:teuthology.orchestra.run.smithi062.stdout: mon client directed command retry = 5 2021-07-10T09:18:49.536 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject socket failures = 5000 2021-07-10T09:18:49.536 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = a9719d92-9687-4432-ad8f-7fcc7e386a67 2021-07-10T09:18:49.536 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.536 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2021-07-10T09:18:49.537 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2021-07-10T09:18:49.537 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.537 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2021-07-10T09:18:49.537 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2021-07-10T09:18:49.537 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.538 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2021-07-10T09:18:49.538 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2021-07-10T09:18:49.538 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.538 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2021-07-10T09:18:49.538 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2021-07-10T09:18:49.539 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2021-07-10T09:18:49.539 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.539 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2021-07-10T09:18:49.539 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2021-07-10T09:18:49.539 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.540 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2021-07-10T09:18:49.540 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.540 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2021-07-10T09:18:49.540 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.540 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2021-07-10T09:18:49.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.541 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2021-07-10T09:18:49.541 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2021-07-10T09:18:49.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.541 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2021-07-10T09:18:49.542 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2021-07-10T09:18:49.542 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2021-07-10T09:18:49.542 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore compression algorithm = snappy 2021-07-10T09:18:49.542 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore compression mode = aggressive 2021-07-10T09:18:49.543 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2021-07-10T09:18:49.543 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2021-07-10T09:18:49.543 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2021-07-10T09:18:49.543 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2021-07-10T09:18:49.543 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2021-07-10T09:18:49.544 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2021-07-10T09:18:49.544 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2021-07-10T09:18:49.544 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2021-07-10T09:18:49.544 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2021-07-10T09:18:49.545 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2021-07-10T09:18:49.545 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2021-07-10T09:18:49.545 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.546 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2021-07-10T09:18:49.546 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2021-07-10T09:18:49.546 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2021-07-10T09:18:49.546 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2021-07-10T09:18:49.546 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2021-07-10T09:18:49.547 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:49.547 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:49.547 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2021-07-10T09:18:49.547 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.547 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2021-07-10T09:18:49.548 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2021-07-10T09:18:49.548 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2021-07-10T09:18:49.548 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2021-07-10T09:18:49.548 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2021-07-10T09:18:49.549 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2021-07-10T09:18:49.549 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2021-07-10T09:18:49.549 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:49.549 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2021-07-10T09:18:49.550 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2021-07-10T09:18:49.550 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:49.550 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.550 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-07-10T09:18:49.550 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2021-07-10T09:18:49.551 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2021-07-10T09:18:49.551 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.551 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2021-07-10T09:18:49.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-07-10T09:18:49.552 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-07-10T09:18:49.552 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-10T09:18:49.552 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2021-07-10T09:18:49.552 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2021-07-10T09:18:49.552 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2021-07-10T09:18:49.553 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2021-07-10T09:18:49.553 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-07-10T09:18:49.553 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-07-10T09:18:49.553 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd = 20 2021-07-10T09:18:49.553 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd_mirror = 15 2021-07-10T09:18:49.554 INFO:teuthology.orchestra.run.smithi062.stdout: log to stderr = False 2021-07-10T09:18:49.554 INFO:teuthology.orchestra.run.smithi062.stdout: rbd default features = 125 2021-07-10T09:18:49.554 INFO:teuthology.orchestra.run.smithi062.stdout: rbd mirror image policy type = simple 2021-07-10T09:18:49.554 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2021-07-10T09:18:49.555 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.0] 2021-07-10T09:18:49.555 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.555 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.555 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.1] 2021-07-10T09:18:49.555 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.556 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.556 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.2] 2021-07-10T09:18:49.556 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.556 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.556 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.3] 2021-07-10T09:18:49.557 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.557 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.558 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2021-07-10T09:18:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: chdir = "" 2021-07-10T09:18:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-07-10T09:18:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: auth supported = cephx 2021-07-10T09:18:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: filestore xattr use omap = true 2021-07-10T09:18:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: mon clock drift allowed = 1.000 2021-07-10T09:18:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: osd crush chooseleaf type = 0 2021-07-10T09:18:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: auth debug = true 2021-07-10T09:18:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on old message = true 2021-07-10T09:18:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on bug = true 2021-07-10T09:18:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: mon max pg per osd = 10000 # >= luminous 2021-07-10T09:18:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: mon pg warn max object skew = 0 2021-07-10T09:18:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: # disable pg_autoscaler by default for new pools 2021-07-10T09:18:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_pg_autoscale_mode = off 2021-07-10T09:18:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default size = 2 2021-07-10T09:18:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow primary affinity = true 2021-07-10T09:18:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow pg remap = true 2021-07-10T09:18:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on legacy crush tunables = false 2021-07-10T09:18:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on crush straw calc version zero = false 2021-07-10T09:18:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on no sortbitwise = false 2021-07-10T09:18:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on osd down out interval zero = false 2021-07-10T09:18:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on too few osds = false 2021-07-10T09:18:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-07-10T09:18:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_no_redundancy = false 2021-07-10T09:18:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: mon_allow_pool_size_one = true 2021-07-10T09:18:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-07-10T09:18:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: osd default data pool replay window = 5 2021-07-10T09:18:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: mon allow pool delete = true 2021-07-10T09:18:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: mon cluster log file level = debug 2021-07-10T09:18:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: debug asserts on shutdown = true 2021-07-10T09:18:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: mon health detail to clog = false 2021-07-10T09:18:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: mon host = 172.21.15.105 2021-07-10T09:18:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: mon client directed command retry = 5 2021-07-10T09:18:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject socket failures = 5000 2021-07-10T09:18:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = a9719d92-9687-4432-ad8f-7fcc7e386a67 2021-07-10T09:18:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.568 INFO:teuthology.orchestra.run.smithi105.stdout:[osd] 2021-07-10T09:18:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: osd journal size = 100 2021-07-10T09:18:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub load threshold = 5.0 2021-07-10T09:18:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub max interval = 600 2021-07-10T09:18:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: osd recover clone overlap = true 2021-07-10T09:18:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: osd recovery max chunk = 1048576 2021-07-10T09:18:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug shutdown = true 2021-07-10T09:18:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug op order = true 2021-07-10T09:18:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify stray on activate = true 2021-07-10T09:18:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: osd open classes on start = true 2021-07-10T09:18:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug pg log writeout = true 2021-07-10T09:18:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: osd deep scrub update digest min age = 30 2021-07-10T09:18:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: osd map max advance = 10 2021-07-10T09:18:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: journal zero on create = true 2021-07-10T09:18:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: filestore ondisk finisher threads = 3 2021-07-10T09:18:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: filestore apply finisher threads = 3 2021-07-10T09:18:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: bdev debug aio = true 2021-07-10T09:18:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug misdirected ops = true 2021-07-10T09:18:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore block size = 96636764160 2021-07-10T09:18:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression algorithm = snappy 2021-07-10T09:18:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression mode = aggressive 2021-07-10T09:18:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore fsck on mount = True 2021-07-10T09:18:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluefs = 1/20 2021-07-10T09:18:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluestore = 1/20 2021-07-10T09:18:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2021-07-10T09:18:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: debug osd = 20 2021-07-10T09:18:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: debug rocksdb = 4/10 2021-07-10T09:18:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd backfillfull_ratio = 0.85 2021-07-10T09:18:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd full ratio = 0.9 2021-07-10T09:18:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd nearfull ratio = 0.8 2021-07-10T09:18:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: osd failsafe full ratio = 0.95 2021-07-10T09:18:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: osd objectstore = bluestore 2021-07-10T09:18:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.578 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr] 2021-07-10T09:18:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2021-07-10T09:18:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: debug mgr = 20 2021-07-10T09:18:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2021-07-10T09:18:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2021-07-10T09:18:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: mgr/telemetry/nag = false 2021-07-10T09:18:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.580 INFO:teuthology.orchestra.run.smithi105.stdout:[mon] 2021-07-10T09:18:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2021-07-10T09:18:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2021-07-10T09:18:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: debug paxos = 20 2021-07-10T09:18:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2021-07-10T09:18:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: mon data avail warn = 5 2021-07-10T09:18:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: mon mgr mkfs grace = 240 2021-07-10T09:18:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2021-07-10T09:18:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd reporter subtree level = osd 2021-07-10T09:18:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd prime pg temp = true 2021-07-10T09:18:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2021-07-10T09:18:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-07-10T09:18:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: auth mon ticket ttl = 660 # 11m 2021-07-10T09:18:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: auth service ticket ttl = 240 # 4m 2021-07-10T09:18:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: # don't complain about insecure global_id in the test suite 2021-07-10T09:18:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-07-10T09:18:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-07-10T09:18:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:18:49.584 INFO:teuthology.orchestra.run.smithi105.stdout:[client] 2021-07-10T09:18:49.585 INFO:teuthology.orchestra.run.smithi105.stdout: rgw cache enabled = true 2021-07-10T09:18:49.585 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable ops log = true 2021-07-10T09:18:49.585 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable usage log = true 2021-07-10T09:18:49.585 INFO:teuthology.orchestra.run.smithi105.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-07-10T09:18:49.585 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-07-10T09:18:49.586 INFO:teuthology.orchestra.run.smithi105.stdout: debug rbd = 20 2021-07-10T09:18:49.586 INFO:teuthology.orchestra.run.smithi105.stdout: debug rbd_mirror = 15 2021-07-10T09:18:49.586 INFO:teuthology.orchestra.run.smithi105.stdout: log to stderr = False 2021-07-10T09:18:49.586 INFO:teuthology.orchestra.run.smithi105.stdout: rbd default features = 125 2021-07-10T09:18:49.587 INFO:teuthology.orchestra.run.smithi105.stdout: rbd mirror image policy type = simple 2021-07-10T09:18:49.587 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.a] 2021-07-10T09:18:49.587 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.0] 2021-07-10T09:18:49.587 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.587 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.588 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.1] 2021-07-10T09:18:49.588 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.588 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.588 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.2] 2021-07-10T09:18:49.588 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.589 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.589 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.3] 2021-07-10T09:18:49.589 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-07-10T09:18:49.589 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-07-10T09:18:49.591 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2021-07-10T09:18:49.591 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2021-07-10T09:18:49.682 INFO:tasks.ceph:Copying monmap to all nodes... 2021-07-10T09:18:49.683 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:49.683 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2021-07-10T09:18:49.699 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:49.700 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2021-07-10T09:18:49.756 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2021-07-10T09:18:49.756 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:49.757 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/cluster2.keyring 2021-07-10T09:18:49.757 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2021-07-10T09:18:49.799 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:18:49.800 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:18:49.858 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:18:49.859 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:49.860 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/cluster2.keyring 2021-07-10T09:18:49.860 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2021-07-10T09:18:49.895 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:18:49.895 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:18:49.951 INFO:tasks.ceph:Setting up mon nodes... 2021-07-10T09:18:49.952 INFO:tasks.ceph:Setting up mgr nodes... 2021-07-10T09:18:49.952 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2021-07-10T09:18:50.008 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2021-07-10T09:18:50.011 INFO:tasks.ceph:Setting up mds nodes... 2021-07-10T09:18:50.011 INFO:tasks.ceph_client:Setting up client nodes... 2021-07-10T09:18:50.011 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2021-07-10T09:18:50.054 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/cluster2.client.0.keyring 2021-07-10T09:18:50.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2021-07-10T09:18:50.107 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2021-07-10T09:18:50.119 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2021-07-10T09:18:50.199 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2021-07-10T09:18:50.211 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2021-07-10T09:18:50.289 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2021-07-10T09:18:50.301 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2021-07-10T09:18:50.378 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2021-07-10T09:18:50.391 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2021-07-10T09:18:50.470 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2021-07-10T09:18:50.482 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2021-07-10T09:18:50.562 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2021-07-10T09:18:50.574 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2021-07-10T09:18:50.654 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2021-07-10T09:18:50.667 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2021-07-10T09:18:50.746 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2021-07-10T09:18:50.757 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-07-10T09:18:50.758 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-07-10T09:18:50.758 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2021-07-10T09:18:50.823 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:50.823 INFO:tasks.ceph:role: cluster2.osd.0 2021-07-10T09:18:50.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:18:50.824 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-07-10T09:18:50.895 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-07-10T09:18:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2021-07-10T09:18:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-07-10T09:18:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 2021-07-10T09:18:50.896 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-07-10T09:18:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2021-07-10T09:18:50.896 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-07-10T09:18:50.897 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-07-10T09:18:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-07-10T09:18:50.897 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-07-10T09:18:51.270 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2021-07-10T09:18:51.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2021-07-10T09:18:51.271 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2021-07-10T09:18:51.407 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2021-07-10T09:18:51.437 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2021-07-10T09:18:51.504 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:51.505 INFO:tasks.ceph:role: cluster2.osd.1 2021-07-10T09:18:51.507 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:18:51.507 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-07-10T09:18:51.576 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-07-10T09:18:51.577 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2021-07-10T09:18:51.577 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-07-10T09:18:51.578 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 2021-07-10T09:18:51.578 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-07-10T09:18:51.578 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2021-07-10T09:18:51.578 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-07-10T09:18:51.579 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-07-10T09:18:51.579 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-07-10T09:18:51.579 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-07-10T09:18:51.930 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2021-07-10T09:18:51.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2021-07-10T09:18:51.932 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2021-07-10T09:18:52.019 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2021-07-10T09:18:52.048 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2021-07-10T09:18:52.113 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-07-10T09:18:52.113 INFO:tasks.ceph:role: cluster2.osd.2 2021-07-10T09:18:52.114 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:18:52.114 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-07-10T09:18:52.187 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-07-10T09:18:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2021-07-10T09:18:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-07-10T09:18:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 2021-07-10T09:18:52.188 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-07-10T09:18:52.188 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2021-07-10T09:18:52.189 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-07-10T09:18:52.189 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-07-10T09:18:52.189 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-07-10T09:18:52.193 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-07-10T09:18:52.551 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2021-07-10T09:18:52.552 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2021-07-10T09:18:52.553 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2021-07-10T09:18:52.657 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2021-07-10T09:18:52.686 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:18:52.779 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:52.777+0000 7f311fbb60c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2021-07-10T09:18:52.779 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:52.778+0000 7f311fbb60c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2021-07-10T09:18:52.780 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:52.778+0000 7f311fbb60c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2021-07-10T09:18:52.780 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:52.778+0000 7f311fbb60c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2021-07-10T09:18:52.780 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:52.778+0000 7f311fbb60c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2021-07-10T09:18:52.783 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:52.781+0000 7f311fbb60c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2021-07-10T09:18:55.648 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2021-07-10T09:18:55.681 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:18:55.767 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:55.766+0000 7fe0e87840c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2021-07-10T09:18:55.768 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:55.766+0000 7fe0e87840c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2021-07-10T09:18:55.768 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:55.766+0000 7fe0e87840c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2021-07-10T09:18:55.768 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:55.766+0000 7fe0e87840c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2021-07-10T09:18:55.768 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:55.766+0000 7fe0e87840c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2021-07-10T09:18:55.769 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:55.766+0000 7fe0e87840c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2021-07-10T09:18:58.617 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2021-07-10T09:18:58.649 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:18:58.737 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:58.734+0000 7f3e17b200c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2021-07-10T09:18:58.737 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:58.734+0000 7f3e17b200c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2021-07-10T09:18:58.738 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:58.734+0000 7f3e17b200c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2021-07-10T09:18:58.738 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:58.734+0000 7f3e17b200c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2021-07-10T09:18:58.739 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:58.734+0000 7f3e17b200c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2021-07-10T09:18:58.739 INFO:teuthology.orchestra.run.smithi105.stderr:2021-07-10T09:18:58.735+0000 7f3e17b200c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2021-07-10T09:19:01.589 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2021-07-10T09:19:01.620 INFO:tasks.ceph:Reading keys from all nodes... 2021-07-10T09:19:01.621 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:19:01.621 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2021-07-10T09:19:01.656 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:01.656 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2021-07-10T09:19:01.690 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:01.690 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2021-07-10T09:19:01.761 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:01.762 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2021-07-10T09:19:01.830 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2021-07-10T09:19:01.830 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2021-07-10T09:19:01.851 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:01.851 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2021-07-10T09:19:01.888 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:01.889 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2021-07-10T09:19:01.948 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:01.949 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2021-07-10T09:19:02.008 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:02.008 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2021-07-10T09:19:02.065 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:02.065 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2021-07-10T09:19:02.122 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:02.122 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2021-07-10T09:19:02.179 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:02.180 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2021-07-10T09:19:02.237 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:02.237 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2021-07-10T09:19:02.293 INFO:tasks.ceph:Adding keys to all mons... 2021-07-10T09:19:02.294 DEBUG:teuthology.orchestra.run.smithi105:> sudo tee -a /etc/ceph/cluster2.keyring 2021-07-10T09:19:02.359 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.x] 2021-07-10T09:19:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6Zelg8AF/ABAAiQ3lwMfJaTTlXPcZeILIFg== 2021-07-10T09:19:02.360 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.0] 2021-07-10T09:19:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD8ZelgTX1jLhAAX4AtCQ6i+AwOXqW0Mzt4Cg== 2021-07-10T09:19:02.360 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.1] 2021-07-10T09:19:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD/Zelgc3yvLRAAoeHMwN8dSCAYY6xOLAnoVQ== 2021-07-10T09:19:02.361 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.2] 2021-07-10T09:19:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQACZulgBY3OKxAA4k/i3xB3Ycb+jet46Ebcjg== 2021-07-10T09:19:02.361 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2021-07-10T09:19:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6ZelgOMI6AxAACsLlM1ytVAPwXn1eKUaaGg== 2021-07-10T09:19:02.362 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror] 2021-07-10T09:19:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6Zelg9VVQBhAACbwhE+OnypRv1zVqkelIJw== 2021-07-10T09:19:02.362 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.0] 2021-07-10T09:19:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6ZelgXC7XCxAAMoTYN9WullvxZQJ8+J2kUQ== 2021-07-10T09:19:02.363 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.1] 2021-07-10T09:19:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6ZelgolozERAAYFTrfFCe/i0biSwejCkzdA== 2021-07-10T09:19:02.363 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.2] 2021-07-10T09:19:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6ZelgTxyMFhAA/Yzci5NM8NGdApxwZO+cYA== 2021-07-10T09:19:02.364 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.3] 2021-07-10T09:19:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6ZelgK0b9GxAAnR48J7LrWcMDFol81+kubg== 2021-07-10T09:19:02.364 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.4] 2021-07-10T09:19:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6ZelgxBp6IRAADfNOTShe4t0H4izN27OCKQ== 2021-07-10T09:19:02.365 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.5] 2021-07-10T09:19:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6Zelg6mzrJhAAJSpLZm1TzWxQ0e/y0+/PNw== 2021-07-10T09:19:02.365 INFO:teuthology.orchestra.run.smithi105.stdout:[client.mirror.6] 2021-07-10T09:19:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD6ZelgHBpgLBAAqUqFiVNUn4lLurneiKq/aw== 2021-07-10T09:19:02.367 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-07-10T09:19:02.456 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-07-10T09:19:02.506 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-07-10T09:19:02.553 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-07-10T09:19:02.600 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:02.651 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:02.701 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:02.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:02.802 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:02.853 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:02.903 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:02.953 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:03.004 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-07-10T09:19:03.054 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-07-10T09:19:03.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2021-07-10T09:19:03.079 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2021-07-10T09:19:03.174 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2021-07-10T09:19:03.200 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2021-07-10T09:19:03.255 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2021-07-10T09:19:03.256 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2021-07-10T09:19:03.256 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2021-07-10T09:19:03.298 INFO:tasks.ceph.cluster2.mon.a:Started 2021-07-10T09:19:03.299 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2021-07-10T09:19:03.299 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2021-07-10T09:19:03.299 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2021-07-10T09:19:03.306 INFO:tasks.ceph.cluster2.mgr.x:Started 2021-07-10T09:19:03.307 INFO:tasks.ceph:Setting crush tunables to default 2021-07-10T09:19:03.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster cluster2 osd crush tunables default 2021-07-10T09:19:03.391 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: vendor '' 2021-07-10T09:19:03.391 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:03.392 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:03.392 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:03.392 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:03.394 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: vendor '' 2021-07-10T09:19:03.395 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:03.395 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:03.395 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:03.395 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:03.504 INFO:teuthology.orchestra.run.smithi105.stderr:adjusted tunables profile to default 2021-07-10T09:19:03.517 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2021-07-10T09:19:03.517 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:03.518 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2021-07-10T09:19:03.544 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:03.545 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2021-07-10T09:19:03.608 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:19:03.609 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2021-07-10T09:19:03.673 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster cluster2 osd new 048499cc-fd14-4478-b645-b8bc131dbf7b 0 2021-07-10T09:19:03.912 INFO:teuthology.orchestra.run.smithi105.stdout:0 2021-07-10T09:19:03.924 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster cluster2 osd new 5913c108-c9f7-4445-a4a6-28a137f772cc 1 2021-07-10T09:19:04.121 INFO:teuthology.orchestra.run.smithi105.stdout:1 2021-07-10T09:19:04.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster cluster2 osd new 1e86815a-a12c-4933-94a8-1512f61b9a0c 2 2021-07-10T09:19:04.330 INFO:teuthology.orchestra.run.smithi105.stdout:2 2021-07-10T09:19:04.342 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2021-07-10T09:19:04.343 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2021-07-10T09:19:04.345 INFO:tasks.ceph.cluster2.osd.0:Started 2021-07-10T09:19:04.346 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2021-07-10T09:19:04.346 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2021-07-10T09:19:04.348 INFO:tasks.ceph.cluster2.osd.1:Started 2021-07-10T09:19:04.349 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2021-07-10T09:19:04.349 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2021-07-10T09:19:04.352 INFO:tasks.ceph.cluster2.osd.2:Started 2021-07-10T09:19:04.352 INFO:tasks.ceph:Waiting for OSDs to come up 2021-07-10T09:19:04.659 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2021-07-10T09:19:04.837 INFO:teuthology.misc.health.smithi105.stdout: 2021-07-10T09:19:04.837 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":5,"fsid":"a9719d92-9687-4432-ad8f-7fcc7e386a67","created":"2021-07-10T09:19:03.394799+0000","modified":"2021-07-10T09:19:04.328281+0000","last_up_change":"0.000000","last_in_change":"2021-07-10T09:19:04.328281+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"048499cc-fd14-4478-b645-b8bc131dbf7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5913c108-c9f7-4445-a4a6-28a137f772cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e86815a-a12c-4933-94a8-1512f61b9a0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:19:04.847 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-07-10T09:19:04.935 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:19:04.932+0000 7f259eded0c0 -1 Falling back to public interface 2021-07-10T09:19:04.942 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:19:04.940+0000 7f513cbb70c0 -1 Falling back to public interface 2021-07-10T09:19:04.946 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:19:04.944+0000 7f6a110550c0 -1 Falling back to public interface 2021-07-10T09:19:08.403 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:19:08.401+0000 7f259eded0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-07-10T09:19:08.429 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:19:08.427+0000 7f6a110550c0 -1 osd.2 0 log_to_monitors {default=true} 2021-07-10T09:19:08.446 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:19:08.443+0000 7f513cbb70c0 -1 osd.1 0 log_to_monitors {default=true} 2021-07-10T09:19:11.152 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2021-07-10T09:19:11.572 INFO:teuthology.misc.health.smithi105.stdout: 2021-07-10T09:19:11.572 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":7,"fsid":"a9719d92-9687-4432-ad8f-7fcc7e386a67","created":"2021-07-10T09:19:03.394799+0000","modified":"2021-07-10T09:19:10.380281+0000","last_up_change":"0.000000","last_in_change":"2021-07-10T09:19:04.328281+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"048499cc-fd14-4478-b645-b8bc131dbf7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5913c108-c9f7-4445-a4a6-28a137f772cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e86815a-a12c-4933-94a8-1512f61b9a0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:19:11.589 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-07-10T09:19:15.671 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:19:15.669+0000 7f6a077ce700 -1 osd.2 0 waiting for initial osdmap 2021-07-10T09:19:15.677 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:19:15.675+0000 7f6a0016f700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-07-10T09:19:15.689 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: vendor '' 2021-07-10T09:19:15.689 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:15.690 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:15.690 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:15.690 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:15.767 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:19:15.765+0000 7f5133330700 -1 osd.1 0 waiting for initial osdmap 2021-07-10T09:19:15.772 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:19:15.770+0000 7f512bcd1700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-07-10T09:19:15.773 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: vendor '' 2021-07-10T09:19:15.774 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:15.774 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:15.775 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:15.775 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:15.807 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:19:15.804+0000 7f2596d69700 -1 osd.0 0 waiting for initial osdmap 2021-07-10T09:19:15.812 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:19:15.810+0000 7f258df07700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-07-10T09:19:15.812 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: vendor '' 2021-07-10T09:19:15.813 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:15.813 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:15.813 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:15.813 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:17.893 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2021-07-10T09:19:18.175 INFO:teuthology.misc.health.smithi105.stdout: 2021-07-10T09:19:18.175 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":9,"fsid":"a9719d92-9687-4432-ad8f-7fcc7e386a67","created":"2021-07-10T09:19:03.394799+0000","modified":"2021-07-10T09:19:17.402193+0000","last_up_change":"2021-07-10T09:19:16.397385+0000","last_in_change":"2021-07-10T09:19:04.328281+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"048499cc-fd14-4478-b645-b8bc131dbf7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":32333},{"type":"v1","addr":"172.21.15.105:6801","nonce":32333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":32333},{"type":"v1","addr":"172.21.15.105:6803","nonce":32333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":32333},{"type":"v1","addr":"172.21.15.105:6807","nonce":32333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":32333},{"type":"v1","addr":"172.21.15.105:6805","nonce":32333}]},"public_addr":"172.21.15.105:6801/32333","cluster_addr":"172.21.15.105:6803/32333","heartbeat_back_addr":"172.21.15.105:6807/32333","heartbeat_front_addr":"172.21.15.105:6805/32333","state":["exists","up"]},{"osd":1,"uuid":"5913c108-c9f7-4445-a4a6-28a137f772cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":32334},{"type":"v1","addr":"172.21.15.105:6809","nonce":32334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":32334},{"type":"v1","addr":"172.21.15.105:6811","nonce":32334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":32334},{"type":"v1","addr":"172.21.15.105:6815","nonce":32334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":32334},{"type":"v1","addr":"172.21.15.105:6813","nonce":32334}]},"public_addr":"172.21.15.105:6809/32334","cluster_addr":"172.21.15.105:6811/32334","heartbeat_back_addr":"172.21.15.105:6815/32334","heartbeat_front_addr":"172.21.15.105:6813/32334","state":["exists","up"]},{"osd":2,"uuid":"1e86815a-a12c-4933-94a8-1512f61b9a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":32335},{"type":"v1","addr":"172.21.15.105:6817","nonce":32335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":32335},{"type":"v1","addr":"172.21.15.105:6819","nonce":32335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":32335},{"type":"v1","addr":"172.21.15.105:6823","nonce":32335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":32335},{"type":"v1","addr":"172.21.15.105:6821","nonce":32335}]},"public_addr":"172.21.15.105:6817/32335","cluster_addr":"172.21.15.105:6819/32335","heartbeat_back_addr":"172.21.15.105:6823/32335","heartbeat_front_addr":"172.21.15.105:6821/32335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.354926+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.421829+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:19:18.188 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-07-10T09:19:18.188 INFO:tasks.ceph:Creating RBD pool 2021-07-10T09:19:18.188 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster cluster2 osd pool create rbd 8 2021-07-10T09:19:19.419 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'rbd' created 2021-07-10T09:19:19.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-07-10T09:19:19.553 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: vendor '' 2021-07-10T09:19:19.554 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:19.554 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:19.554 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:19.554 INFO:tasks.ceph.cluster2.osd.0.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:19.683 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: vendor '' 2021-07-10T09:19:19.683 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:19.684 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:19.684 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:19.684 INFO:tasks.ceph.cluster2.osd.1.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:19.755 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: vendor '' 2021-07-10T09:19:19.755 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:19.755 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:19.756 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:19.756 INFO:tasks.ceph.cluster2.osd.2.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:19.827 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: vendor '' 2021-07-10T09:19:19.827 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: model 'INTEL SSDPEDMD400G4' 2021-07-10T09:19:19.827 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: serial_short 'PHFT620400TR400BGN' 2021-07-10T09:19:19.828 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: scsi_serial '' 2021-07-10T09:19:19.828 INFO:tasks.ceph.cluster2.mon.a.smithi105.stdout: serial 'INTEL SSDPEDMD400G4_PHFT620400TR400BGN' 2021-07-10T09:19:20.425 INFO:teuthology.orchestra.run.smithi105.stderr:enabled application 'rbd' on pool 'rbd' 2021-07-10T09:19:20.440 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2021-07-10T09:19:20.442 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2021-07-10T09:19:20.443 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-07-10T09:19:20.730 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:19:20.731 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":12,"fsid":"a9719d92-9687-4432-ad8f-7fcc7e386a67","created":"2021-07-10T09:19:03.394799+0000","modified":"2021-07-10T09:19:20.414040+0000","last_up_change":"2021-07-10T09:19:16.397385+0000","last_in_change":"2021-07-10T09:19:04.328281+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-07-10T09:19:17.861076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-10T09:19:18.505044+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"048499cc-fd14-4478-b645-b8bc131dbf7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":32333},{"type":"v1","addr":"172.21.15.105:6801","nonce":32333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":32333},{"type":"v1","addr":"172.21.15.105:6803","nonce":32333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":32333},{"type":"v1","addr":"172.21.15.105:6807","nonce":32333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":32333},{"type":"v1","addr":"172.21.15.105:6805","nonce":32333}]},"public_addr":"172.21.15.105:6801/32333","cluster_addr":"172.21.15.105:6803/32333","heartbeat_back_addr":"172.21.15.105:6807/32333","heartbeat_front_addr":"172.21.15.105:6805/32333","state":["exists","up"]},{"osd":1,"uuid":"5913c108-c9f7-4445-a4a6-28a137f772cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":32334},{"type":"v1","addr":"172.21.15.105:6809","nonce":32334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":32334},{"type":"v1","addr":"172.21.15.105:6811","nonce":32334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":32334},{"type":"v1","addr":"172.21.15.105:6815","nonce":32334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":32334},{"type":"v1","addr":"172.21.15.105:6813","nonce":32334}]},"public_addr":"172.21.15.105:6809/32334","cluster_addr":"172.21.15.105:6811/32334","heartbeat_back_addr":"172.21.15.105:6815/32334","heartbeat_front_addr":"172.21.15.105:6813/32334","state":["exists","up"]},{"osd":2,"uuid":"1e86815a-a12c-4933-94a8-1512f61b9a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":32335},{"type":"v1","addr":"172.21.15.105:6817","nonce":32335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":32335},{"type":"v1","addr":"172.21.15.105:6819","nonce":32335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":32335},{"type":"v1","addr":"172.21.15.105:6823","nonce":32335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":32335},{"type":"v1","addr":"172.21.15.105:6821","nonce":32335}]},"public_addr":"172.21.15.105:6817/32335","cluster_addr":"172.21.15.105:6819/32335","heartbeat_back_addr":"172.21.15.105:6823/32335","heartbeat_front_addr":"172.21.15.105:6821/32335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.354926+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.465560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.421829+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:19:20.744 INFO:tasks.ceph.ceph_manager.cluster2:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2021-07-10T09:19:17.861076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-07-10T09:19:18.505044+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-07-10T09:19:20.745 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd pool get .mgr pg_num 2021-07-10T09:19:21.027 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2021-07-10T09:19:21.043 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd pool get rbd pg_num 2021-07-10T09:19:21.323 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 8 2021-07-10T09:19:21.336 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2021-07-10T09:19:21.336 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2021-07-10T09:19:21.337 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2021-07-10T09:19:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:19:21.674 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":5,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":35858},{"type":"v1","addr":"172.21.15.62:6827","nonce":35858}]},"active_addr":"172.21.15.62:6827/35858","active_change":"2021-07-10T09:19:07.607248+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":3423144896}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":362649946}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":3831821186}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2502809305}]}]}} 2021-07-10T09:19:21.677 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2021-07-10T09:19:21.677 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2021-07-10T09:19:21.677 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2021-07-10T09:19:21.955 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:19:21.955 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":13,"fsid":"a9719d92-9687-4432-ad8f-7fcc7e386a67","created":"2021-07-10T09:19:03.394799+0000","modified":"2021-07-10T09:19:21.416995+0000","last_up_change":"2021-07-10T09:19:16.397385+0000","last_in_change":"2021-07-10T09:19:04.328281+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-07-10T09:19:17.861076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-10T09:19:18.505044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"048499cc-fd14-4478-b645-b8bc131dbf7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":32333},{"type":"v1","addr":"172.21.15.105:6801","nonce":32333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":32333},{"type":"v1","addr":"172.21.15.105:6803","nonce":32333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":32333},{"type":"v1","addr":"172.21.15.105:6807","nonce":32333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":32333},{"type":"v1","addr":"172.21.15.105:6805","nonce":32333}]},"public_addr":"172.21.15.105:6801/32333","cluster_addr":"172.21.15.105:6803/32333","heartbeat_back_addr":"172.21.15.105:6807/32333","heartbeat_front_addr":"172.21.15.105:6805/32333","state":["exists","up"]},{"osd":1,"uuid":"5913c108-c9f7-4445-a4a6-28a137f772cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":32334},{"type":"v1","addr":"172.21.15.105:6809","nonce":32334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":32334},{"type":"v1","addr":"172.21.15.105:6811","nonce":32334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":32334},{"type":"v1","addr":"172.21.15.105:6815","nonce":32334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":32334},{"type":"v1","addr":"172.21.15.105:6813","nonce":32334}]},"public_addr":"172.21.15.105:6809/32334","cluster_addr":"172.21.15.105:6811/32334","heartbeat_back_addr":"172.21.15.105:6815/32334","heartbeat_front_addr":"172.21.15.105:6813/32334","state":["exists","up"]},{"osd":2,"uuid":"1e86815a-a12c-4933-94a8-1512f61b9a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":32335},{"type":"v1","addr":"172.21.15.105:6817","nonce":32335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":32335},{"type":"v1","addr":"172.21.15.105:6819","nonce":32335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":32335},{"type":"v1","addr":"172.21.15.105:6823","nonce":32335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":32335},{"type":"v1","addr":"172.21.15.105:6821","nonce":32335}]},"public_addr":"172.21.15.105:6817/32335","cluster_addr":"172.21.15.105:6819/32335","heartbeat_back_addr":"172.21.15.105:6823/32335","heartbeat_front_addr":"172.21.15.105:6821/32335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.354926+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.465560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.421829+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:19:21.968 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2021-07-10T09:19:21.968 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2021-07-10T09:19:22.246 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:19:22.247 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":13,"fsid":"a9719d92-9687-4432-ad8f-7fcc7e386a67","created":"2021-07-10T09:19:03.394799+0000","modified":"2021-07-10T09:19:21.416995+0000","last_up_change":"2021-07-10T09:19:16.397385+0000","last_in_change":"2021-07-10T09:19:04.328281+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-07-10T09:19:17.861076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-07-10T09:19:18.505044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"048499cc-fd14-4478-b645-b8bc131dbf7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":32333},{"type":"v1","addr":"172.21.15.105:6801","nonce":32333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":32333},{"type":"v1","addr":"172.21.15.105:6803","nonce":32333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":32333},{"type":"v1","addr":"172.21.15.105:6807","nonce":32333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":32333},{"type":"v1","addr":"172.21.15.105:6805","nonce":32333}]},"public_addr":"172.21.15.105:6801/32333","cluster_addr":"172.21.15.105:6803/32333","heartbeat_back_addr":"172.21.15.105:6807/32333","heartbeat_front_addr":"172.21.15.105:6805/32333","state":["exists","up"]},{"osd":1,"uuid":"5913c108-c9f7-4445-a4a6-28a137f772cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":32334},{"type":"v1","addr":"172.21.15.105:6809","nonce":32334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":32334},{"type":"v1","addr":"172.21.15.105:6811","nonce":32334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":32334},{"type":"v1","addr":"172.21.15.105:6815","nonce":32334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":32334},{"type":"v1","addr":"172.21.15.105:6813","nonce":32334}]},"public_addr":"172.21.15.105:6809/32334","cluster_addr":"172.21.15.105:6811/32334","heartbeat_back_addr":"172.21.15.105:6815/32334","heartbeat_front_addr":"172.21.15.105:6813/32334","state":["exists","up"]},{"osd":2,"uuid":"1e86815a-a12c-4933-94a8-1512f61b9a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":32335},{"type":"v1","addr":"172.21.15.105:6817","nonce":32335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":32335},{"type":"v1","addr":"172.21.15.105:6819","nonce":32335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":32335},{"type":"v1","addr":"172.21.15.105:6823","nonce":32335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":32335},{"type":"v1","addr":"172.21.15.105:6821","nonce":32335}]},"public_addr":"172.21.15.105:6817/32335","cluster_addr":"172.21.15.105:6819/32335","heartbeat_back_addr":"172.21.15.105:6823/32335","heartbeat_front_addr":"172.21.15.105:6821/32335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.354926+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.465560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-07-10T09:19:09.421829+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-07-10T09:19:22.260 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2021-07-10T09:19:22.408 INFO:teuthology.orchestra.run.smithi105.stdout:34359738376 2021-07-10T09:19:22.408 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2021-07-10T09:19:22.554 INFO:teuthology.orchestra.run.smithi105.stdout:34359738378 2021-07-10T09:19:22.555 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2021-07-10T09:19:22.701 INFO:teuthology.orchestra.run.smithi105.stdout:34359738377 2021-07-10T09:19:22.702 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2021-07-10T09:19:22.981 INFO:teuthology.orchestra.run.smithi105.stdout:34359738375 2021-07-10T09:19:22.995 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.0 2021-07-10T09:19:23.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2021-07-10T09:19:24.288 INFO:teuthology.orchestra.run.smithi105.stdout:34359738375 2021-07-10T09:19:24.301 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.0 2021-07-10T09:19:25.302 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2021-07-10T09:19:25.596 INFO:teuthology.orchestra.run.smithi105.stdout:34359738377 2021-07-10T09:19:25.609 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738377 for osd.0 2021-07-10T09:19:25.610 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2021-07-10T09:19:25.889 INFO:teuthology.orchestra.run.smithi105.stdout:34359738378 2021-07-10T09:19:25.903 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738378 got 34359738378 for osd.1 2021-07-10T09:19:25.903 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2021-07-10T09:19:26.182 INFO:teuthology.orchestra.run.smithi105.stdout:34359738377 2021-07-10T09:19:26.195 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738377 got 34359738377 for osd.2 2021-07-10T09:19:26.196 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2021-07-10T09:19:26.196 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2021-07-10T09:19:26.470 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:19:26.472 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2021-07-10T09:19:26.483 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-07-10T09:19:25.613455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":52552,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":51456,"kb_avail":283062968,"statfs":{"total":289910292480,"available":289856479232,"internally_reserved":0,"allocated":966656,"data_stored":1272683,"data_compressed":60062,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":52690944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000352"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433102+0000","last_change":"2021-07-10T09:19:20.428623+0000","last_active":"2021-07-10T09:19:20.433102+0000","last_peered":"2021-07-10T09:19:20.433102+0000","last_clean":"2021-07-10T09:19:20.433102+0000","last_became_active":"2021-07-10T09:19:20.428346+0000","last_became_peered":"2021-07-10T09:19:20.428346+0000","last_unstale":"2021-07-10T09:19:20.433102+0000","last_undegraded":"2021-07-10T09:19:20.433102+0000","last_fullsized":"2021-07-10T09:19:20.433102+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.435619+0000","last_change":"2021-07-10T09:19:20.430906+0000","last_active":"2021-07-10T09:19:20.435619+0000","last_peered":"2021-07-10T09:19:20.435619+0000","last_clean":"2021-07-10T09:19:20.435619+0000","last_became_active":"2021-07-10T09:19:20.430578+0000","last_became_peered":"2021-07-10T09:19:20.430578+0000","last_unstale":"2021-07-10T09:19:20.435619+0000","last_undegraded":"2021-07-10T09:19:20.435619+0000","last_fullsized":"2021-07-10T09:19:20.435619+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433087+0000","last_change":"2021-07-10T09:19:20.428609+0000","last_active":"2021-07-10T09:19:20.433087+0000","last_peered":"2021-07-10T09:19:20.433087+0000","last_clean":"2021-07-10T09:19:20.433087+0000","last_became_active":"2021-07-10T09:19:20.428309+0000","last_became_peered":"2021-07-10T09:19:20.428309+0000","last_unstale":"2021-07-10T09:19:20.433087+0000","last_undegraded":"2021-07-10T09:19:20.433087+0000","last_fullsized":"2021-07-10T09:19:20.433087+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.430407+0000","last_change":"2021-07-10T09:19:20.426215+0000","last_active":"2021-07-10T09:19:20.430407+0000","last_peered":"2021-07-10T09:19:20.430407+0000","last_clean":"2021-07-10T09:19:20.430407+0000","last_became_active":"2021-07-10T09:19:20.425959+0000","last_became_peered":"2021-07-10T09:19:20.425959+0000","last_unstale":"2021-07-10T09:19:20.430407+0000","last_undegraded":"2021-07-10T09:19:20.430407+0000","last_fullsized":"2021-07-10T09:19:20.430407+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.424696+0000","last_change":"2021-07-10T09:19:19.419843+0000","last_active":"2021-07-10T09:19:20.424696+0000","last_peered":"2021-07-10T09:19:20.424696+0000","last_clean":"2021-07-10T09:19:20.424696+0000","last_became_active":"2021-07-10T09:19:19.418791+0000","last_became_peered":"2021-07-10T09:19:19.418791+0000","last_unstale":"2021-07-10T09:19:20.424696+0000","last_undegraded":"2021-07-10T09:19:20.424696+0000","last_fullsized":"2021-07-10T09:19:20.424696+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:18.407505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:18.407505+0000","last_clean_scrub_stamp":"2021-07-10T09:19:18.407505+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433032+0000","last_change":"2021-07-10T09:19:20.428102+0000","last_active":"2021-07-10T09:19:20.433032+0000","last_peered":"2021-07-10T09:19:20.433032+0000","last_clean":"2021-07-10T09:19:20.433032+0000","last_became_active":"2021-07-10T09:19:20.427781+0000","last_became_peered":"2021-07-10T09:19:20.427781+0000","last_unstale":"2021-07-10T09:19:20.433032+0000","last_undegraded":"2021-07-10T09:19:20.433032+0000","last_fullsized":"2021-07-10T09:19:20.433032+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433021+0000","last_change":"2021-07-10T09:19:20.427949+0000","last_active":"2021-07-10T09:19:20.433021+0000","last_peered":"2021-07-10T09:19:20.433021+0000","last_clean":"2021-07-10T09:19:20.433021+0000","last_became_active":"2021-07-10T09:19:20.427639+0000","last_became_peered":"2021-07-10T09:19:20.427639+0000","last_unstale":"2021-07-10T09:19:20.433021+0000","last_undegraded":"2021-07-10T09:19:20.433021+0000","last_fullsized":"2021-07-10T09:19:20.433021+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.426354+0000","last_change":"2021-07-10T09:19:20.426354+0000","last_active":"2021-07-10T09:19:20.426354+0000","last_peered":"2021-07-10T09:19:20.426354+0000","last_clean":"2021-07-10T09:19:20.426354+0000","last_became_active":"2021-07-10T09:19:20.426091+0000","last_became_peered":"2021-07-10T09:19:20.426091+0000","last_unstale":"2021-07-10T09:19:20.426354+0000","last_undegraded":"2021-07-10T09:19:20.426354+0000","last_fullsized":"2021-07-10T09:19:20.426354+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433128+0000","last_change":"2021-07-10T09:19:20.429086+0000","last_active":"2021-07-10T09:19:20.433128+0000","last_peered":"2021-07-10T09:19:20.433128+0000","last_clean":"2021-07-10T09:19:20.433128+0000","last_became_active":"2021-07-10T09:19:20.428890+0000","last_became_peered":"2021-07-10T09:19:20.428890+0000","last_unstale":"2021-07-10T09:19:20.433128+0000","last_undegraded":"2021-07-10T09:19:20.433128+0000","last_fullsized":"2021-07-10T09:19:20.433128+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":1180736,"data_compressed":60062,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17016,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":16896,"kb_avail":94354824,"statfs":{"total":96636764160,"available":96619339776,"internally_reserved":0,"allocated":114688,"data_stored":31810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17301504},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17764,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354076,"statfs":{"total":96636764160,"available":96618573824,"internally_reserved":0,"allocated":421888,"data_stored":618695,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17772,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354068,"statfs":{"total":96636764160,"available":96618565632,"internally_reserved":0,"allocated":430080,"data_stored":622178,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-07-10T09:19:26.485 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2021-07-10T09:19:26.764 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:19:26.766 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2021-07-10T09:19:26.777 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-07-10T09:19:25.613455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":52552,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":51456,"kb_avail":283062968,"statfs":{"total":289910292480,"available":289856479232,"internally_reserved":0,"allocated":966656,"data_stored":1272683,"data_compressed":60062,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":52690944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000352"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433102+0000","last_change":"2021-07-10T09:19:20.428623+0000","last_active":"2021-07-10T09:19:20.433102+0000","last_peered":"2021-07-10T09:19:20.433102+0000","last_clean":"2021-07-10T09:19:20.433102+0000","last_became_active":"2021-07-10T09:19:20.428346+0000","last_became_peered":"2021-07-10T09:19:20.428346+0000","last_unstale":"2021-07-10T09:19:20.433102+0000","last_undegraded":"2021-07-10T09:19:20.433102+0000","last_fullsized":"2021-07-10T09:19:20.433102+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.435619+0000","last_change":"2021-07-10T09:19:20.430906+0000","last_active":"2021-07-10T09:19:20.435619+0000","last_peered":"2021-07-10T09:19:20.435619+0000","last_clean":"2021-07-10T09:19:20.435619+0000","last_became_active":"2021-07-10T09:19:20.430578+0000","last_became_peered":"2021-07-10T09:19:20.430578+0000","last_unstale":"2021-07-10T09:19:20.435619+0000","last_undegraded":"2021-07-10T09:19:20.435619+0000","last_fullsized":"2021-07-10T09:19:20.435619+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433087+0000","last_change":"2021-07-10T09:19:20.428609+0000","last_active":"2021-07-10T09:19:20.433087+0000","last_peered":"2021-07-10T09:19:20.433087+0000","last_clean":"2021-07-10T09:19:20.433087+0000","last_became_active":"2021-07-10T09:19:20.428309+0000","last_became_peered":"2021-07-10T09:19:20.428309+0000","last_unstale":"2021-07-10T09:19:20.433087+0000","last_undegraded":"2021-07-10T09:19:20.433087+0000","last_fullsized":"2021-07-10T09:19:20.433087+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.430407+0000","last_change":"2021-07-10T09:19:20.426215+0000","last_active":"2021-07-10T09:19:20.430407+0000","last_peered":"2021-07-10T09:19:20.430407+0000","last_clean":"2021-07-10T09:19:20.430407+0000","last_became_active":"2021-07-10T09:19:20.425959+0000","last_became_peered":"2021-07-10T09:19:20.425959+0000","last_unstale":"2021-07-10T09:19:20.430407+0000","last_undegraded":"2021-07-10T09:19:20.430407+0000","last_fullsized":"2021-07-10T09:19:20.430407+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.424696+0000","last_change":"2021-07-10T09:19:19.419843+0000","last_active":"2021-07-10T09:19:20.424696+0000","last_peered":"2021-07-10T09:19:20.424696+0000","last_clean":"2021-07-10T09:19:20.424696+0000","last_became_active":"2021-07-10T09:19:19.418791+0000","last_became_peered":"2021-07-10T09:19:19.418791+0000","last_unstale":"2021-07-10T09:19:20.424696+0000","last_undegraded":"2021-07-10T09:19:20.424696+0000","last_fullsized":"2021-07-10T09:19:20.424696+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:18.407505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:18.407505+0000","last_clean_scrub_stamp":"2021-07-10T09:19:18.407505+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433032+0000","last_change":"2021-07-10T09:19:20.428102+0000","last_active":"2021-07-10T09:19:20.433032+0000","last_peered":"2021-07-10T09:19:20.433032+0000","last_clean":"2021-07-10T09:19:20.433032+0000","last_became_active":"2021-07-10T09:19:20.427781+0000","last_became_peered":"2021-07-10T09:19:20.427781+0000","last_unstale":"2021-07-10T09:19:20.433032+0000","last_undegraded":"2021-07-10T09:19:20.433032+0000","last_fullsized":"2021-07-10T09:19:20.433032+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433021+0000","last_change":"2021-07-10T09:19:20.427949+0000","last_active":"2021-07-10T09:19:20.433021+0000","last_peered":"2021-07-10T09:19:20.433021+0000","last_clean":"2021-07-10T09:19:20.433021+0000","last_became_active":"2021-07-10T09:19:20.427639+0000","last_became_peered":"2021-07-10T09:19:20.427639+0000","last_unstale":"2021-07-10T09:19:20.433021+0000","last_undegraded":"2021-07-10T09:19:20.433021+0000","last_fullsized":"2021-07-10T09:19:20.433021+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.426354+0000","last_change":"2021-07-10T09:19:20.426354+0000","last_active":"2021-07-10T09:19:20.426354+0000","last_peered":"2021-07-10T09:19:20.426354+0000","last_clean":"2021-07-10T09:19:20.426354+0000","last_became_active":"2021-07-10T09:19:20.426091+0000","last_became_peered":"2021-07-10T09:19:20.426091+0000","last_unstale":"2021-07-10T09:19:20.426354+0000","last_undegraded":"2021-07-10T09:19:20.426354+0000","last_fullsized":"2021-07-10T09:19:20.426354+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-07-10T09:19:20.433128+0000","last_change":"2021-07-10T09:19:20.429086+0000","last_active":"2021-07-10T09:19:20.433128+0000","last_peered":"2021-07-10T09:19:20.433128+0000","last_clean":"2021-07-10T09:19:20.433128+0000","last_became_active":"2021-07-10T09:19:20.428890+0000","last_became_peered":"2021-07-10T09:19:20.428890+0000","last_unstale":"2021-07-10T09:19:20.433128+0000","last_undegraded":"2021-07-10T09:19:20.433128+0000","last_fullsized":"2021-07-10T09:19:20.433128+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-07-10T09:19:19.409550+0000","last_clean_scrub_stamp":"2021-07-10T09:19:19.409550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":1180736,"data_compressed":60062,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17016,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":16896,"kb_avail":94354824,"statfs":{"total":96636764160,"available":96619339776,"internally_reserved":0,"allocated":114688,"data_stored":31810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17301504},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17764,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354076,"statfs":{"total":96636764160,"available":96618573824,"internally_reserved":0,"allocated":421888,"data_stored":618695,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17772,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":94354068,"statfs":{"total":96636764160,"available":96618565632,"internally_reserved":0,"allocated":430080,"data_stored":622178,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":590368,"data_compressed":30031,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-07-10T09:19:26.778 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2021-07-10T09:19:26.779 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2021-07-10T09:19:26.779 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2021-07-10T09:19:26.780 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2021-07-10T09:19:27.117 INFO:teuthology.orchestra.run.smithi105.stdout: 2021-07-10T09:19:27.118 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-07-10T09:19:27.131 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2021-07-10T09:19:27.131 INFO:teuthology.run_tasks:Running task exec... 2021-07-10T09:19:27.144 INFO:teuthology.task.exec:Executing custom commands... 2021-07-10T09:19:27.144 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi105.front.sepia.ceph.com 2021-07-10T09:19:27.145 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:27.496 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror 2021-07-10T09:19:27.508 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:27.863 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.0 2021-07-10T09:19:27.874 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.1 2021-07-10T09:19:28.236 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:28.590 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.2 2021-07-10T09:19:28.601 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:28.951 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.3 2021-07-10T09:19:28.963 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:29.315 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror 2021-07-10T09:19:29.327 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:29.681 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.0 2021-07-10T09:19:29.693 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:30.047 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.1 2021-07-10T09:19:30.058 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:30.410 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.2 2021-07-10T09:19:30.422 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-07-10T09:19:30.774 INFO:teuthology.orchestra.run.smithi105.stderr:updated caps for client.mirror.3 2021-07-10T09:19:30.786 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:30.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2021-07-10T09:19:30.814 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2021-07-10T09:19:30.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2021-07-10T09:19:30.817 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:30.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2021-07-10T09:19:30.857 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2021-07-10T09:19:30.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2021-07-10T09:19:30.860 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:30.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2021-07-10T09:19:30.923 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2021-07-10T09:19:30.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2021-07-10T09:19:30.927 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:30.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2021-07-10T09:19:30.992 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2021-07-10T09:19:30.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2021-07-10T09:19:30.995 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:31.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:19:31.058 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:19:31.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:19:31.061 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:31.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2021-07-10T09:19:31.124 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2021-07-10T09:19:31.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2021-07-10T09:19:31.127 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:31.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2021-07-10T09:19:31.191 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2021-07-10T09:19:31.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2021-07-10T09:19:31.193 INFO:teuthology.run_tasks:Running task rbd-mirror... 2021-07-10T09:19:31.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2021-07-10T09:19:31.258 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2021-07-10T09:19:31.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2021-07-10T09:19:31.261 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2021-07-10T09:19:31.326 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1625908771 2021-07-10T09:19:31.326 DEBUG:tasks.rbd_mirror_thrash:Yielding 2021-07-10T09:19:31.327 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2021-07-10T09:19:31.390 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1625908771 2021-07-10T09:19:31.391 DEBUG:tasks.rbd_mirror_thrash:Yielding 2021-07-10T09:19:31.391 INFO:teuthology.run_tasks:Running task workunit... 2021-07-10T09:19:31.463 INFO:tasks.workunit:Pulling workunits from ref 92adad3411b7280fb159eb9d7304c654695beb8b 2021-07-10T09:19:31.464 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-10T09:19:31.465 DEBUG:teuthology.orchestra.run.smithi105:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2021-07-10T09:19:31.465 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2021-07-10T09:19:31.466 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2021-07-10T09:19:31.466 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2021-07-10T09:19:31.466 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2021-07-10T09:19:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:19:31.483 INFO:teuthology.orchestra.run.smithi105.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2021-07-10T09:19:31.484 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2021-07-10T09:19:31.539 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2021-07-10T09:19:31.540 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2021-07-10T09:19:31.595 INFO:tasks.workunit:timeout=6h 2021-07-10T09:19:31.595 INFO:tasks.workunit:cleanup=True 2021-07-10T09:19:31.596 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 92adad3411b7280fb159eb9d7304c654695beb8b 2021-07-10T09:19:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2021-07-10T09:20:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Note: switching to '92adad3411b7280fb159eb9d7304c654695beb8b'. 2021-07-10T09:20:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:20:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-07-10T09:20:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:changes and commit them, and you can discard any commits you make in this 2021-07-10T09:20:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:state without impacting any branches by switching back to a branch. 2021-07-10T09:20:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:20:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:If you want to create a new branch to retain commits you create, you may 2021-07-10T09:20:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:do so (now or later) by using -c with the switch command. Example: 2021-07-10T09:20:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:20:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: git switch -c 2021-07-10T09:20:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:20:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Or undo this operation with: 2021-07-10T09:20:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:20:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: git switch - 2021-07-10T09:20:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:20:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-07-10T09:20:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:20:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:HEAD is now at 92adad3411b rbd-mirror: fix mirror image removal by MrFreezeex · Pull Request #41696 · ceph/ceph · GitHub 2021-07-10T09:20:05.332 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2021-07-10T09:20:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-07-10T09:20:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2021-07-10T09:20:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-07-10T09:20:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-07-10T09:20:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-07-10T09:20:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2021-07-10T09:20:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2021-07-10T09:20:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-07-10T09:20:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2021-07-10T09:20:06.126 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2021-07-10T09:20:06.126 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2021-07-10T09:20:06.184 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2021-07-10T09:20:06.186 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2021-07-10T09:20:06.186 DEBUG:teuthology.orchestra.run.smithi105:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=92adad3411b7280fb159eb9d7304c654695beb8b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2021-07-10T09:20:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ IMAGE_COUNT=50 2021-07-10T09:20:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ export LOCKDEP=0 2021-07-10T09:20:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ LOCKDEP=0 2021-07-10T09:20:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2021-07-10T09:20:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2021-07-10T09:20:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ type xmlstarlet 2021-07-10T09:20:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ XMLSTARLET=xmlstarlet 2021-07-10T09:20:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ RBD_MIRROR_INSTANCES=4 2021-07-10T09:20:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CLUSTER1=cluster1 2021-07-10T09:20:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CLUSTER2=cluster2 2021-07-10T09:20:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ PEER_CLUSTER_SUFFIX= 2021-07-10T09:20:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ POOL=mirror 2021-07-10T09:20:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ PARENT_POOL=mirror_parent 2021-07-10T09:20:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ NS1=ns1 2021-07-10T09:20:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ NS2=ns2 2021-07-10T09:20:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ TEMPDIR= 2021-07-10T09:20:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_ID=mirror 2021-07-10T09:20:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ RBD_IMAGE_FEATURES=layering 2021-07-10T09:20:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2021-07-10T09:20:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ MIRROR_POOL_MODE=image 2021-07-10T09:20:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ MIRROR_IMAGE_MODE=snapshot 2021-07-10T09:20:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ export 'CEPH_ARGS=--id mirror' 2021-07-10T09:20:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_ARGS='--id mirror' 2021-07-10T09:20:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ LAST_MIRROR_INSTANCE=3 2021-07-10T09:20:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2021-07-10T09:20:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2021-07-10T09:20:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2021-07-10T09:20:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_BIN=. 2021-07-10T09:20:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_SRC=. 2021-07-10T09:20:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' -e CMakeCache.txt ']' 2021-07-10T09:20:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 -gt 0 ']' 2021-07-10T09:20:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ setup 2021-07-10T09:20:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local c 2021-07-10T09:20:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ trap 'cleanup $?' INT TERM EXIT 2021-07-10T09:20:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ setup_tempdir 2021-07-10T09:20:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mktemp -d 2021-07-10T09:20:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ TEMPDIR=/tmp/tmp.VSH8FeaLBn 2021-07-10T09:20:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -z 1 ']' 2021-07-10T09:20:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ setup_pools cluster1 cluster2 2021-07-10T09:20:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local remote_cluster=cluster2 2021-07-10T09:20:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mon_map_file 2021-07-10T09:20:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mon_addr 2021-07-10T09:20:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local admin_key_file 2021-07-10T09:20:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local uuid 2021-07-10T09:20:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2021-07-10T09:20:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:pool 'mirror' already exists 2021-07-10T09:20:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2021-07-10T09:20:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:pool 'mirror_parent' already exists 2021-07-10T09:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 pool init mirror 2021-07-10T09:20:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2021-07-10T09:20:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2021-07-10T09:20:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2021-07-10T09:20:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2021-07-10T09:20:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2021-07-10T09:20:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2021-07-10T09:20:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2021-07-10T09:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -z ']' 2021-07-10T09:20:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -z ']' 2021-07-10T09:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_add cluster1 mirror cluster2 2021-07-10T09:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local client_cluster=cluster2 2021-07-10T09:20:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local remote_cluster=cluster2 2021-07-10T09:20:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local uuid_var_name 2021-07-10T09:20:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local error_code 2021-07-10T09:20:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local peer_uuid 2021-07-10T09:20:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 16 32 2021-07-10T09:20:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:20:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2021-07-10T09:20:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_uuid=7a624ece-9f4a-4fd7-a784-9488003f21f4 2021-07-10T09:20:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=0 2021-07-10T09:20:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:20:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 0 -eq 17 ']' 2021-07-10T09:20:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 0 -eq 0 2021-07-10T09:20:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_add cluster1 mirror_parent cluster2 2021-07-10T09:20:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror_parent 2021-07-10T09:20:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local client_cluster=cluster2 2021-07-10T09:20:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local remote_cluster=cluster2 2021-07-10T09:20:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local uuid_var_name 2021-07-10T09:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local error_code 2021-07-10T09:20:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local peer_uuid 2021-07-10T09:20:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 16 32 2021-07-10T09:20:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:20:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2021-07-10T09:20:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_uuid=bb364c85-a1ac-4b52-bf78-0b074f4f9408 2021-07-10T09:20:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=0 2021-07-10T09:20:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:20:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 0 -eq 17 ']' 2021-07-10T09:20:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 0 -eq 0 2021-07-10T09:20:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ setup_pools cluster2 cluster1 2021-07-10T09:20:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local remote_cluster=cluster1 2021-07-10T09:20:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mon_map_file 2021-07-10T09:20:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mon_addr 2021-07-10T09:20:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local admin_key_file 2021-07-10T09:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local uuid 2021-07-10T09:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2021-07-10T09:20:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:pool 'mirror' already exists 2021-07-10T09:20:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2021-07-10T09:20:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:pool 'mirror_parent' already exists 2021-07-10T09:20:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 pool init mirror 2021-07-10T09:20:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2021-07-10T09:20:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2021-07-10T09:20:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2021-07-10T09:20:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2021-07-10T09:20:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2021-07-10T09:20:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2021-07-10T09:20:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2021-07-10T09:20:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -z ']' 2021-07-10T09:20:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -z ']' 2021-07-10T09:20:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_add cluster2 mirror cluster1 2021-07-10T09:20:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local client_cluster=cluster1 2021-07-10T09:20:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local remote_cluster=cluster1 2021-07-10T09:20:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local uuid_var_name 2021-07-10T09:20:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local error_code 2021-07-10T09:20:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local peer_uuid 2021-07-10T09:20:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 16 32 2021-07-10T09:20:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:20:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2021-07-10T09:20:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_uuid=fb875e24-38b4-42ab-8cf6-09827feafdfb 2021-07-10T09:20:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=0 2021-07-10T09:20:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:20:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 0 -eq 17 ']' 2021-07-10T09:20:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 0 -eq 0 2021-07-10T09:20:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_add cluster2 mirror_parent cluster1 2021-07-10T09:20:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror_parent 2021-07-10T09:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local client_cluster=cluster1 2021-07-10T09:20:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local remote_cluster=cluster1 2021-07-10T09:20:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local uuid_var_name 2021-07-10T09:20:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local error_code 2021-07-10T09:20:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local peer_uuid 2021-07-10T09:20:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 16 32 2021-07-10T09:20:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:20:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2021-07-10T09:20:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ peer_uuid=f81f3699-89b4-4109-b61c-01824b918115 2021-07-10T09:20:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=0 2021-07-10T09:20:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:20:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 0 -eq 17 ']' 2021-07-10T09:20:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 0 -eq 0 2021-07-10T09:20:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirrors cluster1 2021-07-10T09:20:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 0 3 2021-07-10T09:20:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster1:0 2021-07-10T09:20:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1:0 2021-07-10T09:20:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:20:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster1:0 2021-07-10T09:20:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster1 2021-07-10T09:20:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=0 2021-07-10T09:20:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster1 2021-07-10T09:20:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=0 2021-07-10T09:20:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:20:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster1:1 2021-07-10T09:20:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1:1 2021-07-10T09:20:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:20:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster1:1 2021-07-10T09:20:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster1 2021-07-10T09:20:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=1 2021-07-10T09:20:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster1 2021-07-10T09:20:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=1 2021-07-10T09:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster1:2 2021-07-10T09:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1:2 2021-07-10T09:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster1:2 2021-07-10T09:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster1 2021-07-10T09:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=2 2021-07-10T09:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster1 2021-07-10T09:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=2 2021-07-10T09:20:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:20:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster1:3 2021-07-10T09:20:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1:3 2021-07-10T09:20:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:20:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster1:3 2021-07-10T09:20:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster1 2021-07-10T09:20:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=3 2021-07-10T09:20:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster1 2021-07-10T09:20:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=3 2021-07-10T09:20:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:20:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirrors cluster2 2021-07-10T09:20:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 0 3 2021-07-10T09:20:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster2:0 2021-07-10T09:20:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2:0 2021-07-10T09:20:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-07-10T09:20:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster2:0 2021-07-10T09:20:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster2 2021-07-10T09:20:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=0 2021-07-10T09:20:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 0 = cluster2:0 ']' 2021-07-10T09:20:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster2 2021-07-10T09:20:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster2 2021-07-10T09:20:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=0 2021-07-10T09:20:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:20:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster2:1 2021-07-10T09:20:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2:1 2021-07-10T09:20:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-07-10T09:20:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster2:1 2021-07-10T09:20:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster2 2021-07-10T09:20:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=1 2021-07-10T09:20:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 1 = cluster2:1 ']' 2021-07-10T09:20:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster2 2021-07-10T09:20:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster2 2021-07-10T09:20:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=1 2021-07-10T09:20:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:20:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster2:2 2021-07-10T09:20:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2:2 2021-07-10T09:20:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-07-10T09:20:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster2:2 2021-07-10T09:20:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster2 2021-07-10T09:20:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=2 2021-07-10T09:20:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 2 = cluster2:2 ']' 2021-07-10T09:20:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster2 2021-07-10T09:20:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster2 2021-07-10T09:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=2 2021-07-10T09:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster2:3 2021-07-10T09:20:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2:3 2021-07-10T09:20:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:20:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-07-10T09:20:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster2:3 2021-07-10T09:20:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:20:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:20:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster2 2021-07-10T09:20:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=3 2021-07-10T09:20:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 3 = cluster2:3 ']' 2021-07-10T09:20:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster2 2021-07-10T09:20:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster2 2021-07-10T09:20:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=3 2021-07-10T09:20:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:20:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:20:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:20:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2021-07-10T09:20:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee -a /tmp/tmp.VSH8FeaLBn/rbd-mirror.test.log 2021-07-10T09:20:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ date '+%F %T' 2021-07-10T09:20:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo 2021-07-10 09:20:21 TEST: add image and test replay after client crashes 2021-07-10T09:20:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10 09:20:21 TEST: add image and test replay after client crashes 2021-07-10T09:20:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=test 2021-07-10T09:20:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2021-07-10T09:20:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:20:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:20:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror test 512M 2021-07-10T09:20:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:20:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:20:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:20:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:20:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2021-07-10T09:20:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:20:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror test snapshot 2021-07-10T09:20:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:20:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2021-07-10T09:20:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:20:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:20:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:20:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:20:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:20:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:20:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:20:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:20:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:20:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:20:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:20:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:20:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:20:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:20:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:20:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:20:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:20:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:20:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:20:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:20:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:20:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:20:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:20:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:20:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:20:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:20:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:20:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:20:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:20:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:20:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:20:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:20:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:20:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:20:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:20:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:20:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:20:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:20:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:20:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:20:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:20:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:20:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:20:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:20:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:20:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:20:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:20:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:23.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:20:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:20:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:20:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:20:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:20:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:20:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:20:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:20:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:20:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:20:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:20:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:20:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:20:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:20:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:20:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:20:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:20:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:20:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:20:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:20:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:20:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:20:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:20:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:20:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:20:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:20:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:20:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:20:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:20:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 1 2021-07-10T09:20:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:20:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:20:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:20:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:20:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:20:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:20:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:20:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:20:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:20:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:20:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:20:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:20:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:20:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:20:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:20:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:20:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:20:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:20:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:20:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:20:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:20:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:20:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:20:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:20:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:20:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:20:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:20:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:20:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:20:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:20:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:20:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:20:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:20:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:20:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:20:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:20:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:20:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:20:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:20:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:20:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:20:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:20:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:20:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:20:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:20:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:20:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:20:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:20:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:20:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:20:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:20:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:20:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:20:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:20:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:20:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 2 2021-07-10T09:20:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:20:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:20:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:20:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:20:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:20:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:20:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:20:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:20:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:20:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:20:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:20:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:20:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:20:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:20:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:20:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:20:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:20:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:20:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:20:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:20:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:20:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:20:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:20:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:20:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:20:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:20:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:20:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:20:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:20:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:20:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:20:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:20:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:20:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:20:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:20:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:20:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:20:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:20:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:20:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:20:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:20:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:20:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:20:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:20:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:20:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:20:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:20:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:20:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:20:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:20:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:20:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:20:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:20:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:20:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:20:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 4 2021-07-10T09:20:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:20:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:20:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:20:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:20:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:20:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:20:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:20:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:20:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:20:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:20:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:20:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:20:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:20:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:20:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:20:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:20:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:20:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:20:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:20:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:20:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:20:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:20:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:20:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:20:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:20:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:20:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:20:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:20:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:20:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:20:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:20:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:20:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:20:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:20:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:20:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:20:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:20:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:20:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:20:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:20:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:20:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:20:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:20:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:20:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:20:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:20:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:20:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:20:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:20:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:20:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:20:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:20:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:20:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:20:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:20:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:20:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:20:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:20:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:20:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:20:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:20:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:20:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:20:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:20:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 8 2021-07-10T09:20:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:20:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:20:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:20:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:20:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:20:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:20:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:20:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:20:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:20:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:20:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:20:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:20:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:20:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:20:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:20:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:20:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:20:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:20:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:20:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:20:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:20:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:20:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:20:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:20:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:20:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:20:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:20:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:20:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:20:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:20:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:20:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:20:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name= 2021-07-10T09:20:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 10 2021-07-10T09:20:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:20:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:20:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:20:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:20:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:20:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:20:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 817152 bytes 2021-07-10T09:20:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 15820800 bytes 2021-07-10T09:20:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 35031040 bytes 2021-07-10T09:20:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 37630976 bytes 2021-07-10T09:20:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 61883392 bytes 2021-07-10T09:20:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 102769664 bytes 2021-07-10T09:20:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 212953088 bytes 2021-07-10T09:20:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 246953984 bytes 2021-07-10T09:20:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 293299200 bytes 2021-07-10T09:20:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 323890176 bytes 2021-07-10T09:20:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 378680320 bytes 2021-07-10T09:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 383063040 bytes 2021-07-10T09:20:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 396935168 bytes 2021-07-10T09:20:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 421760000 bytes 2021-07-10T09:20:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 448979968 bytes 2021-07-10T09:20:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 530332672 bytes 2021-07-10T09:20:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 598303744 bytes 2021-07-10T09:20:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 634871808 bytes 2021-07-10T09:20:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 708379648 bytes 2021-07-10T09:20:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 724033536 bytes 2021-07-10T09:20:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 771671040 bytes 2021-07-10T09:20:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 840337408 bytes 2021-07-10T09:20:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 850264064 bytes 2021-07-10T09:20:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 913052672 bytes 2021-07-10T09:20:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 921443328 bytes 2021-07-10T09:20:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 995152896 bytes 2021-07-10T09:20:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1006255104 bytes 2021-07-10T09:20:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1042221056 bytes 2021-07-10T09:20:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1066228736 bytes 2021-07-10T09:20:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1111534592 bytes 2021-07-10T09:20:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1161836544 bytes 2021-07-10T09:20:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1183041536 bytes 2021-07-10T09:20:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1236340736 bytes 2021-07-10T09:20:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1314323456 bytes 2021-07-10T09:20:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1382966272 bytes 2021-07-10T09:20:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1447341056 bytes 2021-07-10T09:20:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1479941120 bytes 2021-07-10T09:20:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1511473152 bytes 2021-07-10T09:20:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1555398656 bytes 2021-07-10T09:20:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1582377984 bytes 2021-07-10T09:20:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1598777344 bytes 2021-07-10T09:20:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1599372288 bytes 2021-07-10T09:20:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1664244736 bytes 2021-07-10T09:20:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1693219840 bytes 2021-07-10T09:20:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1738545152 bytes 2021-07-10T09:20:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 1817948160 bytes 2021-07-10T09:20:47.467 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6820625487879833) > weight (0.25) 2021-07-10T09:20:47.467 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5201749326192462) > weight (0.25) 2021-07-10T09:20:47.467 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6909655281044407) > weight (0.25) 2021-07-10T09:20:47.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8124553132004554) > weight (0.25) 2021-07-10T09:20:47.468 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2021-07-10T09:20:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 1903226880 bytes 2021-07-10T09:20:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4800 IOs, 1903842304 bytes 2021-07-10T09:20:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4900 IOs, 1904455680 bytes 2021-07-10T09:20:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5000 IOs, 1953987584 bytes 2021-07-10T09:20:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5100 IOs, 1979262976 bytes 2021-07-10T09:20:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:20:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:20:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:20:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:20:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:20:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:20:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:20:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:20:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:20:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:20:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:20:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:20:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:20:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:20:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:20:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:20:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:20:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:20:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:20:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"remote_snapshot_timestamp":1625908822,"replay_state":"idle"} 2021-07-10T09:20:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.0 on smithi105 2021-07-10T09:20:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:20:33 2021-07-10T09:20:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:20:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:20:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:20:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:20:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:20:33 2021-07-10T09:20:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:20:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"remote_snapshot_timestamp":1625908822,"replay_state":"idle"} 2021-07-10T09:20:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:20:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:20:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:20:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:20:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.0 on smithi105 2021-07-10T09:20:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:20:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:20:33' 2021-07-10T09:20:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:20:33' 2021-07-10T09:20:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:20:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:20:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:20:33' 2021-07-10T09:20:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:20:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:20:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:20:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:20:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:20:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:20:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:20:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:20:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:20:33' 2021-07-10T09:20:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:20:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:20:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:20:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:20:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:20:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:20:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap1 2021-07-10T09:20:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap1 2021-07-10T09:20:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:20:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:20:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:20:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap1 2021-07-10T09:20:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:21:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:21:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:21:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:21:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:21:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:21:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:21:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:21:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:21:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:21:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:21:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:21:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:21:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:21:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:21:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:21:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:21:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:21:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:21:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:21:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:21:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:21:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:21:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:21:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:21:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:21:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:21:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:21:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:21:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:21:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:21:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:21:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:21:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:21:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:21:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:21:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:21:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:21:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:21:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:21:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:21:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:21:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:21:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:21:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:21:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:21:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:21:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:21:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:21:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:21:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:21:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:21:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:21:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:21:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:21:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:21:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:21:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:21:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:21:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:21:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:21:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:21:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:21:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:21:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:21:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:21:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:21:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:21:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:21:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:21:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:21:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:21:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:21:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:21:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:21:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:21:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:21:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:21:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:21:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:21:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:21:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 6 2021-07-10T09:21:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:21:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:21:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:21:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:21:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:21:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:21:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=6 2021-07-10T09:21:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:21:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:21:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:21:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:21:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:21:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:21:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:21:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:21:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:21:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:21:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:21:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 6 = '' 2021-07-10T09:21:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:21:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:21:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:21:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:21:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:21:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:21:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:21:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:21:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:21:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:21:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 6 = '' 2021-07-10T09:21:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:21:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:21:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:21:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:21:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:21:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:21:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:21:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:21:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:21:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:21:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 6 = '' 2021-07-10T09:21:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:21:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:21:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:21:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:21:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:21:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:21:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=6 2021-07-10T09:21:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 6 = 6 2021-07-10T09:21:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:21:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2021-07-10T09:21:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap1 2021-07-10T09:21:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:21:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:21:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:21:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2021-07-10T09:21:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:21:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:21:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:21:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:21:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:21:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:21:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:21:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:21:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:21:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:21:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:21:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:21:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:21:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:21:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:21:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:21:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:21:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:21:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:21:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:21:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap1 2021-07-10T09:21:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap1 2021-07-10T09:21:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:21:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:21:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:21:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:21:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:21:23.900 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:21:23.899+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:21:24.054 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:21:24.053+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:21:24.054 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:21:24.053+0000 7f5112290700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:21:24.237 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:21:24.235+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:21:24.238 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:21:24.237+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:21:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:21:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:21:24.768 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:21:24.760+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:21:24.857 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:21:24.762+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:21:24.904 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:21:24.902+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:21:24.905 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:21:24.903+0000 7f422a97b700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2021-07-10T09:21:24.905 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:21:24.903+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:21:25.159 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:21:25.158+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:21:25.160 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:21:25.158+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:21:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:21:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:21:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:21:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:21:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap1-clean 2021-07-10T09:21:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap1-clean 2021-07-10T09:21:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:21:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap1-clean 2021-07-10T09:21:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:21:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:21:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:21:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:21:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:21:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=7 2021-07-10T09:21:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:21:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:21:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 34665472 bytes 2021-07-10T09:21:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 55451648 bytes 2021-07-10T09:21:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 76884992 bytes 2021-07-10T09:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 98941952 bytes 2021-07-10T09:21:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 151582720 bytes 2021-07-10T09:21:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 168878080 bytes 2021-07-10T09:21:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 177558528 bytes 2021-07-10T09:21:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 208420864 bytes 2021-07-10T09:21:28.467 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2021-07-10T09:21:28.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2021-07-10T09:21:28.470 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2021-07-10T09:21:28.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi105.stderr:2021-07-10T09:21:28.468+0000 7f46b1c4b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36602) UID: 1000 2021-07-10T09:21:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 217331712 bytes 2021-07-10T09:21:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 326282240 bytes 2021-07-10T09:21:28.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:21:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 339274752 bytes 2021-07-10T09:21:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 347119616 bytes 2021-07-10T09:21:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 364770304 bytes 2021-07-10T09:21:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 420799488 bytes 2021-07-10T09:21:29.471 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2021-07-10T09:21:29.472 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:21:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:21:29.473 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2021-07-10T09:21:29.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2021-07-10T09:21:29.474 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2021-07-10T09:21:29.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2021-07-10T09:21:29.484 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2021-07-10T09:21:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 431260672 bytes 2021-07-10T09:21:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 508460032 bytes 2021-07-10T09:21:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 560371712 bytes 2021-07-10T09:21:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 605867008 bytes 2021-07-10T09:21:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 606486528 bytes 2021-07-10T09:21:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 621733888 bytes 2021-07-10T09:21:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 662122496 bytes 2021-07-10T09:21:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 684104704 bytes 2021-07-10T09:21:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 721292288 bytes 2021-07-10T09:21:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 764472320 bytes 2021-07-10T09:21:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 836909056 bytes 2021-07-10T09:21:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 917534720 bytes 2021-07-10T09:21:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1001518080 bytes 2021-07-10T09:21:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1022163968 bytes 2021-07-10T09:21:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1024890880 bytes 2021-07-10T09:21:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1134659584 bytes 2021-07-10T09:21:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1163266048 bytes 2021-07-10T09:21:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1249025024 bytes 2021-07-10T09:21:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1275547648 bytes 2021-07-10T09:21:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1309067264 bytes 2021-07-10T09:21:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:21:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:21:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:21:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:21:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:21:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:21:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:21:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:21:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:21:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:21:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:21:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:21:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:21:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:21:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:21:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:21:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:21:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:21:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:21:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:21:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:21:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:21:31 2021-07-10T09:21:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:21:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:21:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:21:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:21:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:21:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:21:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:21:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:21:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:21:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:21:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:21:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:21:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:21:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:21:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:21:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:21:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:21:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:21:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:21:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:21:31 2021-07-10T09:21:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:21:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:21:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:21:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:21:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:21:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:21:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:21:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:21:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:21:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:21:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:21:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:21:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:21:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:21:31 2021-07-10T09:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:21:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:21:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:21:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:21:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:21:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:21:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:21:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:21:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:21:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:21:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:21:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:21:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:21:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:21:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:21:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:21:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:21:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:21:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:21:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:21:31 2021-07-10T09:21:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:21:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:21:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:21:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:21:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:21:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:21:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:21:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:21:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:21:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:21:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:21:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:21:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:21:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:21:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:21:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:21:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:21:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:21:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:21:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:21:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:21:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:21:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:21:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:21:31 2021-07-10T09:21:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:21:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:21:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:21:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:22:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:22:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:22:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:22:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:22:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:22:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:22:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:22:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:22:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:22:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:22:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1625908879,"remote_snapshot_timestamp":1625908879,"replay_state":"idle"} 2021-07-10T09:22:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.1 on smithi105 2021-07-10T09:22:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:22:01 2021-07-10T09:22:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:22:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:22:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:22:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:22:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:22:01 2021-07-10T09:22:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:22:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1625908879,"remote_snapshot_timestamp":1625908879,"replay_state":"idle"} 2021-07-10T09:22:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:22:04.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:22:04.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:22:04.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:22:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.1 on smithi105 2021-07-10T09:22:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:22:01' 2021-07-10T09:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:22:01' 2021-07-10T09:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:22:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:22:01' 2021-07-10T09:22:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:22:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:22:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:22:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:22:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:22:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:22:01' 2021-07-10T09:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:22:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:22:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:22:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap2 2021-07-10T09:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap2 2021-07-10T09:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap2 2021-07-10T09:22:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:22:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:22:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:22:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:22:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:22:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:22:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:22:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:22:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:22:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:22:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:22:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:22:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:22:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:22:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:22:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:22:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:22:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:22:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:22:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:22:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:22:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:22:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:22:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:22:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:22:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:22:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:22:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:22:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:22:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:22:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:22:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:22:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:22:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:22:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:22:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:22:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:22:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:22:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:22:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:22:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:22:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:22:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:22:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:22:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:22:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:22:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:22:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:22:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:22:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:22:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:22:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:22:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:22:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:22:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:22:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:22:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:22:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:22:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:22:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:22:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:22:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:22:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:22:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:22:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 10 2021-07-10T09:22:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:22:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:22:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:22:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:22:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:22:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=10 2021-07-10T09:22:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:22:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:22:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:22:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:22:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:22:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:22:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=6 2021-07-10T09:22:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 10 = 6 2021-07-10T09:22:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:22:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:22:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:22:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:22:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:22:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:22:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:22:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:22:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:22:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:22:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 10 = '' 2021-07-10T09:22:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:22:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:22:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:22:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:22:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:22:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:22:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:22:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:22:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:22:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:22:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 10 = '' 2021-07-10T09:22:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:22:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:22:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:22:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:22:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:22:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:22:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:22:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:22:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:22:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:22:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 10 = '' 2021-07-10T09:22:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:22:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:22:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:22:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:22:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:22:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:22:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=10 2021-07-10T09:22:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 10 = 10 2021-07-10T09:22:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2021-07-10T09:22:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap2 2021-07-10T09:22:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:22:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:22:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:22:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2021-07-10T09:22:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:22:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:22:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:22:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 4 2021-07-10T09:22:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:22:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:22:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:22:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:22:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:22:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:22:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:22:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:22:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:22:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:22:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:22:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:22:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:22:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:22:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap1-clean ']' 2021-07-10T09:22:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap1-clean 2021-07-10T09:22:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap1-clean 2021-07-10T09:22:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:22:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:22:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:22:14.014 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:22:14.013+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:22:14.445 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:22:14.443+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2021-07-10T09:22:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:22:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:14.681 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:22:14.679+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:22:15.175 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:22:15.173+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:22:15.176 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:22:15.175+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 8 2021-07-10T09:22:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:22:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap2 2021-07-10T09:22:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap2 2021-07-10T09:22:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:22:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:22:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:22:16.297 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:22:16.295+0000 7f69e572c700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:22:16.297 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:22:16.295+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:22:16.298 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:22:16.296+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:22:16.345 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:22:16.343+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:22:16.346 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:22:16.344+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:22:16.346 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:22:16.344+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:22:16.346 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:22:16.344+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:22:16.347 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:22:16.345+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:22:16.348 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:22:16.345+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:22:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:22:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:16.678 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:22:16.676+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:22:16.678 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:22:16.677+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:22:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:22:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:22:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap2-clean 2021-07-10T09:22:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap2-clean 2021-07-10T09:22:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap2-clean 2021-07-10T09:22:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:22:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:22:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:22:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:22:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:22:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:22:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:22:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 50416640 bytes 2021-07-10T09:22:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 71363584 bytes 2021-07-10T09:22:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 144661504 bytes 2021-07-10T09:22:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 198906880 bytes 2021-07-10T09:22:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 310730752 bytes 2021-07-10T09:22:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 352468992 bytes 2021-07-10T09:22:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 413698048 bytes 2021-07-10T09:22:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 460727296 bytes 2021-07-10T09:22:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 527969280 bytes 2021-07-10T09:22:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 539427840 bytes 2021-07-10T09:22:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 559134720 bytes 2021-07-10T09:22:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 587431936 bytes 2021-07-10T09:22:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 617021440 bytes 2021-07-10T09:22:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 677850112 bytes 2021-07-10T09:22:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 716571648 bytes 2021-07-10T09:22:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 761670656 bytes 2021-07-10T09:22:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 841163776 bytes 2021-07-10T09:22:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 893104128 bytes 2021-07-10T09:22:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 922798080 bytes 2021-07-10T09:22:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 958252032 bytes 2021-07-10T09:22:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:22:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:22:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:22:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:22:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:22:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:22:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:22:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:22:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:22:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:22:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:22:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:22:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:22:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1625908879,"remote_snapshot_timestamp":1625908879,"replay_state":"idle"} 2021-07-10T09:22:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.1 on smithi105 2021-07-10T09:22:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:22:01 2021-07-10T09:22:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:22:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:22:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:22:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:22:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:22:01 2021-07-10T09:22:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:22:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1625908879,"remote_snapshot_timestamp":1625908879,"replay_state":"idle"} 2021-07-10T09:22:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:22:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:22:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:22:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:22:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.1 on smithi105 2021-07-10T09:22:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:22:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:22:01' 2021-07-10T09:22:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:22:01' 2021-07-10T09:22:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:22:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:22:01' 2021-07-10T09:22:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:22:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:22:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:22:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:22:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:22:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:22:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:22:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:22:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:22:01' 2021-07-10T09:22:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:22:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:22:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:22:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:22:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:22:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:22:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap3 2021-07-10T09:22:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap3 2021-07-10T09:22:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:22:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap3 2021-07-10T09:22:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:22:29.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5978031140625706) > weight (0.25) 2021-07-10T09:22:29.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6353548128804916) > weight (0.25) 2021-07-10T09:22:29.469 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2021-07-10T09:22:29.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2021-07-10T09:22:29.470 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2021-07-10T09:22:29.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi105.stderr:2021-07-10T09:22:29.469+0000 7ff392035700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36848) UID: 1000 2021-07-10T09:22:29.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:22:31.472 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2021-07-10T09:22:31.472 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:22:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:22:31.473 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2021-07-10T09:22:31.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2021-07-10T09:22:31.473 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2021-07-10T09:22:31.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2021-07-10T09:22:31.477 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2021-07-10T09:22:36.485 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3325278922737004) > weight (0.25) 2021-07-10T09:22:36.485 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2021-07-10T09:22:36.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2021-07-10T09:22:36.486 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2021-07-10T09:22:36.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:22:36.485+0000 7f76e9060700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36620) UID: 1000 2021-07-10T09:22:36.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:22:38.488 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2021-07-10T09:22:38.488 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:22:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:22:38.489 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2021-07-10T09:22:38.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2021-07-10T09:22:38.489 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2021-07-10T09:22:38.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2021-07-10T09:22:38.493 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2021-07-10T09:22:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:22:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:22:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:22:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:22:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:22:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:22:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:22:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:22:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:22:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:22:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:22:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:22:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:22:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:22:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:22:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:22:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:22:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:22:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:22:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:22:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:22:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:22:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:22:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:22:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:22:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:22:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:22:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:22:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:22:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:22:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:22:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:22:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:22:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:22:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:22:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:22:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:22:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:22:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:22:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:22:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:22:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:22:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:22:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:22:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:22:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:22:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:22:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:22:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:22:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:22:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:22:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:22:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:22:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:22:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:22:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:22:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:22:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:22:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:22:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:22:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:22:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:22:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:22:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:22:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:22:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:22:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:22:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:22:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:22:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:22:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:22:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:22:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:22:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:22:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:22:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:22:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:22:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:22:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 1 2021-07-10T09:22:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:22:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:22:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:22:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:22:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:22:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:22:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:22:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:22:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:22:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:22:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:22:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:22:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:22:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:22:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:22:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:22:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:22:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:22:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:22:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:22:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:22:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:22:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:22:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:22:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:22:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:22:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:22:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:22:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:22:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:22:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:22:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:22:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:22:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:22:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:22:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:22:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:22:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:22:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:22:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:22:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:22:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:22:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:22:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:22:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:22:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:22:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:22:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:22:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:22:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:22:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:22:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:22:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:22:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:22:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:22:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 2 2021-07-10T09:22:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:22:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:22:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:22:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:22:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:22:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:22:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:22:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:22:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:22:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:22:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:22:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:22:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:22:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:22:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:22:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:22:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:22:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:22:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:22:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:22:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:22:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:22:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:22:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:22:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:22:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:22:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:22:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:22:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:22:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:22:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:22:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:22:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:22:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:22:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:22:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:22:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:22:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:22:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:22:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:22:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:22:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:22:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:22:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:22:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:22:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:22:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:22:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:22:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:22:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:22:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:22:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:22:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:22:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:22:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:22:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:22:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:22:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:22:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:22:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:22:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:22:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:22:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:22:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:22:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:22:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:22:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:22:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:22:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:22:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:22:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:22:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:22:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:22:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:22:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:22:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:22:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:22:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:22:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:22:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:22:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:22:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:22:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:22:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:22:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:22:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:22:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:22:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:22:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:22:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:22:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 4 2021-07-10T09:23:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:23:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:23:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:23:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:23:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:23:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:23:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:23:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:23:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:23:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:23:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:23:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:23:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:23:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:23:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:23:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:23:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:23:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:23:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:23:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:23:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:23:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:23:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:23:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:23:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:23:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:23:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:23:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:23:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:23:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:23:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:23:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:23:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:23:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:23:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:23:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:23:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:23:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:23:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:23:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:23:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:23:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:23:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:23:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:23:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:23:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:23:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:23:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:23:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:23:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:23:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:23:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:23:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:23:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:23:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:23:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:23:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:23:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:23:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:23:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:23:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:23:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:23:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:23:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:23:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:23:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:23:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:23:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:23:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:23:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:23:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:23:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:23:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:23:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:23:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:23:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:23:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:23:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:23:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:23:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:23:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:23:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:23:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:23:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:23:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:23:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:23:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:23:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:23:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:23:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:23:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:23:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:23:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:23:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:23:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:23:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:23:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:23:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:23:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:23:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:23:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:23:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:23:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:23:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:23:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:23:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:23:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:23:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:23:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:23:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:23:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:23:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:23:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:23:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:23:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:23:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:23:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:23:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:23:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:23:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:23:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:23:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:23:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:23:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:23:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:23:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:23:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:23:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:23:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:23:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 8 2021-07-10T09:23:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:23:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:23:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:23:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:23:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:23:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:23:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:23:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:23:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:23:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:23:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:23:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:23:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:23:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:23:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:23:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:23:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:23:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:23:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:23:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:23:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:23:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:23:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:23:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:23:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:23:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:23:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:23:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:23:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:23:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:23:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:23:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:23:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:23:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:23:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:23:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:23:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:23:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:23:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:23:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:23:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:23:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:23:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:23:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:23:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:23:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:23:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:23:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:23:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:23:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:23:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:23:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:23:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:23:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:23:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:23:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:23:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:23:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:23:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:23:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:23:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:23:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:23:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:23:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:23:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:23:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:23:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:23:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:23:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:23:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:23:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:23:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:23:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:23:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:23:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:23:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:23:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:23:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:23:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:23:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:23:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:23:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:23:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:23:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:23:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:23:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:23:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:23:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:23:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:23:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:23:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:23:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:23:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:23:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:23:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:23:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:23:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:23:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:23:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:23:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:23:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:23:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:23:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:23:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:23:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:23:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:23:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:23:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:23:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:23:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:23:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:23:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:23:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:23:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:23:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:23:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:23:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:23:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:23:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:23:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:23:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:23:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:23:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:23:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:23:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:23:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:23:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:23:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:23:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:23:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:23:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:23:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:23:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 14 2021-07-10T09:23:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:23:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:23:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:23:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:23:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:23:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:23:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=14 2021-07-10T09:23:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:23:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:23:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:23:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:23:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:23:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:23:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:23:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:23:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:23:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:23:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:23:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 14 = '' 2021-07-10T09:23:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:23:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:23:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:23:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:23:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:23:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:23:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:23:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:23:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:23:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 14 = '' 2021-07-10T09:23:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:23:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:23:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:23:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:23:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:23:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:23:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:23:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:23:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:23:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:23:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 14 = '' 2021-07-10T09:23:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:23:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:23:14.724 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:14.722+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:23:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:23:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:23:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:23:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:23:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:23:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:23:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:23:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 14 = '' 2021-07-10T09:23:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:23:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:23:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:23:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:23:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:23:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:23:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=14 2021-07-10T09:23:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 14 = 14 2021-07-10T09:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2021-07-10T09:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap3 2021-07-10T09:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:23:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2021-07-10T09:23:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:23:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:23:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:23:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 6 2021-07-10T09:23:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:23:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:23:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:23:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:23:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:23:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:23:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:23:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:22:14 2021 2021-07-10T09:23:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:23:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:23:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:23:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:23:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:23:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:23:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap2-clean ']' 2021-07-10T09:23:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap2-clean 2021-07-10T09:23:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap2-clean 2021-07-10T09:23:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:23:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:23:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:23:19.071 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:19.069+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:19.127 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:19.125+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:19.332 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:19.330+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:23:19.338 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:19.331+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:23:19.339 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:19.331+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:23:19.396 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:23:19.394+0000 7f69e3f29700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:23:19.397 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:23:19.395+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:23:19.470 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:19.469+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:23:19.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:19.617 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.614+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:19.617 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.614+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:19.617 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.615+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:19.618 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.615+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:23:19.656 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.654+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:19.656 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.654+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:23:19.657 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.654+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:23:19.657 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.654+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:23:19.657 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.654+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:23:19.686 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.680+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:19.686 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.680+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:19.687 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.681+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:23:19.687 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.681+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:23:19.687 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.681+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:23:19.688 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.682+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:23:19.758 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.756+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:23:19.758 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:19.756+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:23:20.119 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:20.118+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:20.120 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:20.118+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:23:20.120 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:20.118+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:20.266 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:23:20.259+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:23:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:23:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap3 2021-07-10T09:23:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap3 2021-07-10T09:23:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:23:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:23:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:23:20.881 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:20.879+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:20.881 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:20.879+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:20.882 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:20.880+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:20.936 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:20.934+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:21.040 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:21.038+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:21.200 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:23:21.197+0000 7f69e472a700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:23:21.201 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:23:21.198+0000 7f69ea736700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:23:21.201 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:23:21.198+0000 7f69e9734700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:23:21.202 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:23:21.198+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:23:21.335 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:23:21.329+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:23:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:23:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:21.678 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:21.677+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:21.738 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:21.736+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:21.739 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:21.737+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:23:21.811 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:23:21.809+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:23:22.250 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:22.248+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:23:22.251 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:22.249+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:23:22.251 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:22.249+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:23:22.251 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:23:22.250+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 10 2021-07-10T09:23:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:23:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:23:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap3-clean 2021-07-10T09:23:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap3-clean 2021-07-10T09:23:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:23:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap3-clean 2021-07-10T09:23:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:23:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:23:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:23:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:23:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:23:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:23:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:23:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:23:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 72962048 bytes 2021-07-10T09:23:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 141973504 bytes 2021-07-10T09:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 160305152 bytes 2021-07-10T09:23:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 212379648 bytes 2021-07-10T09:23:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 234790912 bytes 2021-07-10T09:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 276592640 bytes 2021-07-10T09:23:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 365355008 bytes 2021-07-10T09:23:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 397549568 bytes 2021-07-10T09:23:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 463921152 bytes 2021-07-10T09:23:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 578106368 bytes 2021-07-10T09:23:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 643152896 bytes 2021-07-10T09:23:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 717698048 bytes 2021-07-10T09:23:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 746242048 bytes 2021-07-10T09:23:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 767499264 bytes 2021-07-10T09:23:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 770985984 bytes 2021-07-10T09:23:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 819269632 bytes 2021-07-10T09:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 834794496 bytes 2021-07-10T09:23:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 875725824 bytes 2021-07-10T09:23:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 895015936 bytes 2021-07-10T09:23:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 953633792 bytes 2021-07-10T09:23:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 985784320 bytes 2021-07-10T09:23:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 1036637184 bytes 2021-07-10T09:23:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1069464576 bytes 2021-07-10T09:23:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1119211520 bytes 2021-07-10T09:23:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1151335424 bytes 2021-07-10T09:23:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1185337344 bytes 2021-07-10T09:23:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1232188416 bytes 2021-07-10T09:23:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1259977728 bytes 2021-07-10T09:23:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1304558592 bytes 2021-07-10T09:23:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1325565952 bytes 2021-07-10T09:23:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1382222848 bytes 2021-07-10T09:23:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1422952448 bytes 2021-07-10T09:23:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1436987392 bytes 2021-07-10T09:23:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1498885120 bytes 2021-07-10T09:23:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1568879616 bytes 2021-07-10T09:23:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1589620736 bytes 2021-07-10T09:23:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1628198912 bytes 2021-07-10T09:23:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1698170880 bytes 2021-07-10T09:23:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1719946240 bytes 2021-07-10T09:23:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1742428160 bytes 2021-07-10T09:23:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1773987840 bytes 2021-07-10T09:23:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1865357312 bytes 2021-07-10T09:23:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1927552000 bytes 2021-07-10T09:23:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1951762432 bytes 2021-07-10T09:23:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 2017275904 bytes 2021-07-10T09:23:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 2079251456 bytes 2021-07-10T09:23:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 2136339456 bytes 2021-07-10T09:23:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4800 IOs, 2148293632 bytes 2021-07-10T09:23:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4900 IOs, 2211629056 bytes 2021-07-10T09:23:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5000 IOs, 2262676480 bytes 2021-07-10T09:23:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5100 IOs, 2317897728 bytes 2021-07-10T09:23:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5200 IOs, 2396443648 bytes 2021-07-10T09:23:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5300 IOs, 2461064192 bytes 2021-07-10T09:23:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5400 IOs, 2479425536 bytes 2021-07-10T09:23:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:23:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:23:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:23:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:23:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:23:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:23:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:23:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:23:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:23:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:23:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:23:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:23:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:23:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:23:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:23:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:23:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:23:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:23:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:23:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":21654937.6,"bytes_per_snapshot":433098752.0,"local_snapshot_timestamp":1625908992,"remote_snapshot_timestamp":1625908992,"replay_state":"idle"} 2021-07-10T09:23:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.2 on smithi105 2021-07-10T09:23:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:23:31 2021-07-10T09:23:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:23:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:23:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:23:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:23:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:23:31 2021-07-10T09:23:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:23:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":21654937.6,"bytes_per_snapshot":433098752.0,"local_snapshot_timestamp":1625908992,"remote_snapshot_timestamp":1625908992,"replay_state":"idle"} 2021-07-10T09:23:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:23:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:23:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:23:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:23:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.2 on smithi105 2021-07-10T09:23:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:23:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:23:31' 2021-07-10T09:23:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:23:31' 2021-07-10T09:23:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:23:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:23:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:23:31' 2021-07-10T09:23:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:23:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:23:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:23:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:23:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:23:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:23:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:23:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:23:31' 2021-07-10T09:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:23:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:23:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:23:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:23:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:23:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:23:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap4 2021-07-10T09:23:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap4 2021-07-10T09:23:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:23:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:23:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:23:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap4 2021-07-10T09:23:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:24:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:24:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:24:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:24:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:24:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:24:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:24:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:24:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:24:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:24:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:24:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:24:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:24:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:24:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:24:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:24:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:24:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:24:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:24:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:24:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:24:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:24:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:24:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:24:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:24:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:24:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:24:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:24:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:24:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:24:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:24:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:24:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:24:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:24:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:24:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:24:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:24:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:24:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:24:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:24:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:24:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:24:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:24:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:24:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:24:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:24:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:24:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:24:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:24:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:24:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:24:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:24:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:24:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:24:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:24:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:24:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:24:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:24:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:24:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:24:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:24:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:24:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:24:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:24:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:24:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:24:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:24:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:24:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:24:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:24:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:24:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:24:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:24:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:24:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:24:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:24:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:24:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:24:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:24:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:24:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:24:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:24:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:24:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:24:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:24:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:24:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:24:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:24:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:24:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:24:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:24:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:24:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:24:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:24:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:24:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:24:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:24:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:24:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:24:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:24:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:24:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:24:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:24:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:24:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:24:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:24:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:24:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:24:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:24:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:24:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:24:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:24:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:24:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:24:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:24:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:24:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:24:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:24:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:24:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:24:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:24:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:24:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:24:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:24:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:24:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:24:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:24:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:24:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:24:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:24:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:24:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:24:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:24:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:24:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:24:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:24:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:24:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:24:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:24:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 18 2021-07-10T09:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:24:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:24:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:24:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:24:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=18 2021-07-10T09:24:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:24:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:07.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:07.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 18 = '' 2021-07-10T09:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:24:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 18 = '' 2021-07-10T09:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:24:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 18 = '' 2021-07-10T09:24:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:24:09.730 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:09.727+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:24:09.731 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:09.727+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:24:09.731 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:09.727+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:24:09.731 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:09.727+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:24:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 18 = '' 2021-07-10T09:24:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:24:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=18 2021-07-10T09:24:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 18 = 18 2021-07-10T09:24:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2021-07-10T09:24:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap4 2021-07-10T09:24:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:24:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:24:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2021-07-10T09:24:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:24:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:24:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:24:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 8 2021-07-10T09:24:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:24:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:24:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:24:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:24:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:24:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:24:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:24:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:23:19 2021 2021-07-10T09:24:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:24:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:24:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:24:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:24:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:24:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:24:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap3-clean ']' 2021-07-10T09:24:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap3-clean 2021-07-10T09:24:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap3-clean 2021-07-10T09:24:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:24:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:24:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:24:13.922 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:13.921+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:24:14.091 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:14.086+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2021-07-10T09:24:14.256 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:24:14.254+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:24:14.314 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:14.310+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:24:14.421 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:14.417+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:24:14.421 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:14.417+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:24:14.491 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:24:14.482+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:24:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:24:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:14.853 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:24:14.851+0000 7f72c7892700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:24:14.855 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:24:14.853+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:24:14.856 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:24:14.854+0000 7f4228977700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:24:15.195 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:24:15.192+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:24:15.274 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:24:15.272+0000 7f72c9095700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:24:15.281 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:24:15.276+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:24:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:24:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap4 2021-07-10T09:24:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap4 2021-07-10T09:24:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:24:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:24:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:24:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:24:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:16.399 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:24:16.396+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:24:16.400 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:24:16.396+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:24:16.400 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:24:16.397+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:24:16.400 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:24:16.397+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:24:16.706 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:24:16.704+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:24:16.707 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:24:16.705+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:24:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:24:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:24:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap4-clean 2021-07-10T09:24:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap4-clean 2021-07-10T09:24:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap4-clean 2021-07-10T09:24:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:24:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:24:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:24:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:24:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=7 2021-07-10T09:24:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:24:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:24:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 84663296 bytes 2021-07-10T09:24:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 143276032 bytes 2021-07-10T09:24:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 222396416 bytes 2021-07-10T09:24:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 257853440 bytes 2021-07-10T09:24:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 287782912 bytes 2021-07-10T09:24:19.477 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3234345367886997) > weight (0.25) 2021-07-10T09:24:19.478 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5106567811790537) > weight (0.25) 2021-07-10T09:24:19.478 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4861036549001616) > weight (0.25) 2021-07-10T09:24:19.479 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9127948158118133) > weight (0.25) 2021-07-10T09:24:19.479 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2021-07-10T09:24:19.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3878147060167536) > weight (0.25) 2021-07-10T09:24:19.494 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2021-07-10T09:24:19.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2021-07-10T09:24:19.495 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2021-07-10T09:24:19.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:24:19.494+0000 7f8678253700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 41222) UID: 1000 2021-07-10T09:24:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 288393216 bytes 2021-07-10T09:24:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 307467264 bytes 2021-07-10T09:24:19.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:24:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 388033536 bytes 2021-07-10T09:24:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 428274688 bytes 2021-07-10T09:24:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 482411520 bytes 2021-07-10T09:24:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 508227584 bytes 2021-07-10T09:24:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 548389888 bytes 2021-07-10T09:24:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 626429952 bytes 2021-07-10T09:24:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 692854784 bytes 2021-07-10T09:24:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 729271296 bytes 2021-07-10T09:24:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 764910592 bytes 2021-07-10T09:24:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 816606208 bytes 2021-07-10T09:24:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 817866752 bytes 2021-07-10T09:24:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 854445056 bytes 2021-07-10T09:24:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 855063552 bytes 2021-07-10T09:24:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 888048640 bytes 2021-07-10T09:24:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 975237120 bytes 2021-07-10T09:24:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1037633536 bytes 2021-07-10T09:24:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1109238784 bytes 2021-07-10T09:24:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1111964672 bytes 2021-07-10T09:24:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1174103040 bytes 2021-07-10T09:24:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1241644032 bytes 2021-07-10T09:24:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1263453184 bytes 2021-07-10T09:24:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:24:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:24:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:24:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:24:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:24:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:24:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:24:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:24:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:24:24.496 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2021-07-10T09:24:24.497 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:24:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:24:24.498 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2021-07-10T09:24:24.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2021-07-10T09:24:24.498 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2021-07-10T09:24:24.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2021-07-10T09:24:24.502 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2021-07-10T09:24:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:24:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:24:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:24:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:24:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:24:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:24:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:24:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:24:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":433098752.0,"local_snapshot_timestamp":1625908992,"remote_snapshot_timestamp":1625908992,"replay_state":"idle"} 2021-07-10T09:24:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.2 on smithi105 2021-07-10T09:24:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:24:01 2021-07-10T09:24:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:24:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:24:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:24:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:24:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:24:01 2021-07-10T09:24:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:24:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":433098752.0,"local_snapshot_timestamp":1625908992,"remote_snapshot_timestamp":1625908992,"replay_state":"idle"} 2021-07-10T09:24:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:24:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:24:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:24:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:24:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.2 on smithi105 2021-07-10T09:24:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:24:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:24:01' 2021-07-10T09:24:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:24:01' 2021-07-10T09:24:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:24:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:24:01' 2021-07-10T09:24:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:24:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:24:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:24:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:24:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:24:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:24:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:24:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:24:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:24:01' 2021-07-10T09:24:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:24:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:24:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:24:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:24:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:24:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap5 2021-07-10T09:24:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap5 2021-07-10T09:24:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap5 2021-07-10T09:24:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:24:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:24:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:24:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:24:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:24:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:24:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:24:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:24:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:24:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:24:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:24:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:24:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:24:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:24:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:24:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:24:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:24:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:24:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:24:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:24:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:24:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:24:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:24:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:24:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:24:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:24:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:24:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:24:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:24:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:24:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:24:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:24:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:24:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:24:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:24:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:24:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:24:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:24:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:24:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:24:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:24:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:24:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:24:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:24:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:24:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:24:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:24:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:24:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:24:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:24:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:24:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:24:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:24:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:24:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:24:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:24:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:24:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:24:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:24:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:24:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:24:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:24:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:24:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:24:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:24:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:24:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:24:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:24:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:24:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:24:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:24:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:24:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:24:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:24:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:24:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:24:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:24:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:24:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:24:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:24:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:24:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:24:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:24:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:24:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:24:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:24:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:24:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:24:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:24:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:24:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:24:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:24:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:24:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:24:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:24:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:24:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:24:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:24:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:24:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:24:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:24:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:24:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:24:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:24:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:24:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:24:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:24:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:24:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:24:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:24:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:24:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:24:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:24:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:24:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:24:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:24:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:24:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:24:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:24:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:24:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:24:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:24:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:24:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:24:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:24:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:24:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:24:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:24:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:24:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:24:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:24:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:24:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:24:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:24:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:24:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:24:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:24:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 22 2021-07-10T09:24:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:24:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:24:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:24:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:24:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:24:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=22 2021-07-10T09:24:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:24:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 22 = '' 2021-07-10T09:24:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:24:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 22 = '' 2021-07-10T09:24:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:24:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 22 = '' 2021-07-10T09:24:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:24:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:24:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:24:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:24:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:24:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:24:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:24:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:24:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:24:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:24:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:24:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:24:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:24:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 22 = '' 2021-07-10T09:24:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:24:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:25:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:25:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:25:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:25:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=22 2021-07-10T09:25:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 22 = 22 2021-07-10T09:25:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2021-07-10T09:25:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap5 2021-07-10T09:25:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:25:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:25:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:25:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2021-07-10T09:25:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:25:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:25:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:25:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 10 2021-07-10T09:25:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:25:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:25:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:25:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:25:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:25:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:25:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:25:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:23:19 2021 2021-07-10T09:25:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:25:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:25:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:25:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:25:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:25:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:25:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap4-clean ']' 2021-07-10T09:25:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap4-clean 2021-07-10T09:25:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap4-clean 2021-07-10T09:25:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:25:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:02.952 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:02.950+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:02.952 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:02.950+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:25:02.953 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:02.950+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:02.953 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:02.951+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:03.360 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:03.356+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:03.360 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:03.356+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:03.360 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:03.356+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:03.360 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:03.357+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap5 2021-07-10T09:25:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap5 2021-07-10T09:25:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:25:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:04.548 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:04.545+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:04.549 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:04.547+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:05.006 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:05.005+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:25:05.066 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:05.064+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:25:05.118 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:05.116+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:25:05.119 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:05.116+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:25:05.119 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:05.116+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:25:05.119 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:05.116+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:25:05.119 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:05.117+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:05.185 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:05.183+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:05.304 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.301+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:25:05.385 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.383+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:25:05.386 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.384+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:25:05.387 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.386+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:25:05.462 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.460+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:25:05.530 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:05.528+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:25:05.531 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:05.529+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:25:05.744 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.743+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:25:05.747 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.744+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:25:05.819 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:05.817+0000 7f4229178700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:25:05.820 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:05.818+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:25:05.820 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:05.818+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:25:05.821 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:05.819+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:25:05.876 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:05.875+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap5-clean 2021-07-10T09:25:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap5-clean 2021-07-10T09:25:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap5-clean 2021-07-10T09:25:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:25:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:25:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:25:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:25:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=9 2021-07-10T09:25:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:25:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:25:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 3681280 bytes 2021-07-10T09:25:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 105014272 bytes 2021-07-10T09:25:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 152503296 bytes 2021-07-10T09:25:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 154503168 bytes 2021-07-10T09:25:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 177156096 bytes 2021-07-10T09:25:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 204819456 bytes 2021-07-10T09:25:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 247797760 bytes 2021-07-10T09:25:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 312801280 bytes 2021-07-10T09:25:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 341191680 bytes 2021-07-10T09:25:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 398024704 bytes 2021-07-10T09:25:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 442754048 bytes 2021-07-10T09:25:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 511460352 bytes 2021-07-10T09:25:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 539126784 bytes 2021-07-10T09:25:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 613713920 bytes 2021-07-10T09:25:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 633315328 bytes 2021-07-10T09:25:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 673644544 bytes 2021-07-10T09:25:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 746830848 bytes 2021-07-10T09:25:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 774722560 bytes 2021-07-10T09:25:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 779158528 bytes 2021-07-10T09:25:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 799933440 bytes 2021-07-10T09:25:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 856596480 bytes 2021-07-10T09:25:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 912559104 bytes 2021-07-10T09:25:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 915293184 bytes 2021-07-10T09:25:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 931353600 bytes 2021-07-10T09:25:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 974838784 bytes 2021-07-10T09:25:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1052009472 bytes 2021-07-10T09:25:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1064179712 bytes 2021-07-10T09:25:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1064803328 bytes 2021-07-10T09:25:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1106459648 bytes 2021-07-10T09:25:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1145196544 bytes 2021-07-10T09:25:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1219958784 bytes 2021-07-10T09:25:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1284482048 bytes 2021-07-10T09:25:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1337230336 bytes 2021-07-10T09:25:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1337837568 bytes 2021-07-10T09:25:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1394285568 bytes 2021-07-10T09:25:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1414275072 bytes 2021-07-10T09:25:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:25:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:25:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:25:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:25:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:25:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:25:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:25:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:25:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:25:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:25:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:25:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:25:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:25:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:25:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:25:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:25:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":482026496.0,"local_snapshot_timestamp":1625909047,"remote_snapshot_timestamp":1625909096,"replay_state":"syncing","seconds_until_synced":258786004515684352,"syncing_percent":0,"syncing_snapshot_timestamp":1625909096} 2021-07-10T09:25:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.2 on smithi105 2021-07-10T09:25:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:25:01 2021-07-10T09:25:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:25:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:25:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:25:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:25:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:25:01 2021-07-10T09:25:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:25:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":482026496.0,"local_snapshot_timestamp":1625909047,"remote_snapshot_timestamp":1625909096,"replay_state":"syncing","seconds_until_synced":258786004515684352,"syncing_percent":0,"syncing_snapshot_timestamp":1625909096} 2021-07-10T09:25:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:25:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:25:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:25:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:25:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.2 on smithi105 2021-07-10T09:25:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:25:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:25:01' 2021-07-10T09:25:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:25:01' 2021-07-10T09:25:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:25:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:25:01' 2021-07-10T09:25:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:25:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:25:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:25:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:25:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:25:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:25:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:25:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:25:01' 2021-07-10T09:25:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:25:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:25:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:25:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:25:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:25:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap6 2021-07-10T09:25:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap6 2021-07-10T09:25:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap6 2021-07-10T09:25:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:25:32.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5021816731740987) > weight (0.25) 2021-07-10T09:25:32.504 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2021-07-10T09:25:32.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2021-07-10T09:25:32.505 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2021-07-10T09:25:32.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:25:32.505+0000 7f78dd702700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 42227) UID: 1000 2021-07-10T09:25:32.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:25:38.480 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.33096458267392126) > weight (0.25) 2021-07-10T09:25:38.481 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.45974360620488763) > weight (0.25) 2021-07-10T09:25:38.482 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2021-07-10T09:25:38.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2021-07-10T09:25:38.483 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2021-07-10T09:25:38.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi105.stderr:2021-07-10T09:25:38.484+0000 7fcbf0173700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 41107) UID: 1000 2021-07-10T09:25:38.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:25:40.506 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2021-07-10T09:25:40.506 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:25:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:25:40.507 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2021-07-10T09:25:40.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2021-07-10T09:25:40.508 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2021-07-10T09:25:40.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2021-07-10T09:25:40.512 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2021-07-10T09:25:44.487 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2021-07-10T09:25:44.487 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:25:44.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:25:44.488 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2021-07-10T09:25:44.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2021-07-10T09:25:44.489 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2021-07-10T09:25:44.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2021-07-10T09:25:44.492 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2021-07-10T09:25:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:25:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:25:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:25:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:25:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:25:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:25:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:25:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:25:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:25:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:25:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:25:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:25:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:25:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:25:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:25:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:25:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:25:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:25:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:25:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:25:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:25:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:25:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:25:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:25:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:25:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:25:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:25:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:25:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:25:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:25:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:25:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:25:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:25:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:25:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:25:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:25:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:25:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:25:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:25:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:25:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:25:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:25:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:25:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:25:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:25:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:25:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:25:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:25:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:25:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:25:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:25:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:25:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:25:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:25:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:25:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:25:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:25:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:25:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:25:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:25:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:25:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:25:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:25:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:25:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:25:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:25:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:25:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:25:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:25:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:25:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:25:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:25:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:25:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:25:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:25:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:25:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:25:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:25:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:25:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:25:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:25:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:25:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:25:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:25:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:25:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:25:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:25:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:25:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:25:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:25:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:25:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:25:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:25:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:25:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:25:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:25:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:25:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:25:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:25:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:25:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:25:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:25:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:25:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:25:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:25:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:25:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:25:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:25:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:25:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:25:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:25:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:25:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:25:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:25:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:25:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:25:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:25:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:25:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:25:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:25:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:25:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:25:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:25:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:25:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:25:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:25:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:25:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:25:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:25:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:25:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:25:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:25:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:25:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:25:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 26 2021-07-10T09:25:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:25:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:25:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:25:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:25:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:25:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=26 2021-07-10T09:25:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:25:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:25:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:25:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:25:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:25:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:25:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:25:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:25:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:25:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:25:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:25:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 26 = '' 2021-07-10T09:25:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:25:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:25:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:25:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:25:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:25:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:25:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:25:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:25:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:25:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:25:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 26 = '' 2021-07-10T09:25:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:25:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:25:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:25:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:25:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:25:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:25:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:25:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:25:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:25:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:25:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 26 = '' 2021-07-10T09:25:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:25:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:25:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:25:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:25:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:25:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:25:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:25:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:25:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:25:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:25:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 26 = '' 2021-07-10T09:25:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:25:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:25:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:25:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:25:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:25:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:25:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=26 2021-07-10T09:25:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 26 = 26 2021-07-10T09:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2021-07-10T09:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:25:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap6 2021-07-10T09:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:25:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:25:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:25:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2021-07-10T09:25:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:25:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:25:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:25:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 12 2021-07-10T09:25:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:25:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:25:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:25:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:25:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:25:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:25:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:25:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:25:03 2021 2021-07-10T09:25:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:25:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:25:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:25:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:25:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:25:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:25:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap5-clean ']' 2021-07-10T09:25:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap5-clean 2021-07-10T09:25:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap5-clean 2021-07-10T09:25:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:25:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:55.504 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:55.502+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:25:55.505 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:55.503+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:25:55.505 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:55.503+0000 7f69e472a700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:25:55.564 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:55.562+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:25:55.565 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:55.562+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2021-07-10T09:25:55.565 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:55.563+0000 7f69e8f33700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 7 2021-07-10T09:25:55.642 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:55.639+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:55.642 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:55.640+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:25:55.643 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:55.640+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:25:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:55.791 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.788+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:25:55.792 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.789+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:25:55.792 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.789+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:25:55.797 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.790+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:25:55.915 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.914+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:25:55.983 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.981+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:55.983 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.981+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:25:55.984 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:55.983+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 9 2021-07-10T09:25:56.057 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:56.055+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:25:56.058 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:56.056+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:25:56.058 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:56.056+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:25:56.270 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:56.268+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:56.271 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:56.269+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:25:56.272 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:56.270+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap6 2021-07-10T09:25:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap6 2021-07-10T09:25:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:25:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:25:57.143 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.141+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:57.143 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.141+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:25:57.428 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:57.427+0000 7f69e572c700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2021-07-10T09:25:57.481 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.480+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:25:57.482 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.480+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:57.603 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.601+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:57.603 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.601+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:25:57.604 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.602+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:25:57.604 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:25:57.602+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2021-07-10T09:25:57.661 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:25:57.660+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:25:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:57.882 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.880+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:25:57.882 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.881+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:25:57.883 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.882+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.882+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.883+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:25:57.954 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.952+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:25:57.954 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.952+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:57.955 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:57.952+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:25:58.023 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.021+0000 7f4228977700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:25:58.024 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.021+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:25:58.024 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.022+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:25:58.024 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.022+0000 7f422a17a700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:25:58.152 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.150+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:25:58.152 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.150+0000 7f4226973700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:25:58.153 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.152+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2021-07-10T09:25:58.153 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.152+0000 7f4229178700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2021-07-10T09:25:58.154 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.152+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 7 2021-07-10T09:25:58.221 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:58.219+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:25:58.221 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:58.219+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:25:58.221 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:25:58.219+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:25:58.301 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.298+0000 7f4226973700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:25:58.301 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:25:58.299+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:25:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:25:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:25:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:25:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap6-clean 2021-07-10T09:25:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap6-clean 2021-07-10T09:25:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap6-clean 2021-07-10T09:25:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:25:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:25:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:25:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:25:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:25:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:25:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:25:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:25:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:25:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:26:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 31766528 bytes 2021-07-10T09:26:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 69000192 bytes 2021-07-10T09:26:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 141790208 bytes 2021-07-10T09:26:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 144640000 bytes 2021-07-10T09:26:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 187568128 bytes 2021-07-10T09:26:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 212706304 bytes 2021-07-10T09:26:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 247983104 bytes 2021-07-10T09:26:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 293468160 bytes 2021-07-10T09:26:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 328192000 bytes 2021-07-10T09:26:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 328811520 bytes 2021-07-10T09:26:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 400007168 bytes 2021-07-10T09:26:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 458230784 bytes 2021-07-10T09:26:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 489119744 bytes 2021-07-10T09:26:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 499942400 bytes 2021-07-10T09:26:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 503133184 bytes 2021-07-10T09:26:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 606631936 bytes 2021-07-10T09:26:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 619504640 bytes 2021-07-10T09:26:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 625639424 bytes 2021-07-10T09:26:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 670840832 bytes 2021-07-10T09:26:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 713574400 bytes 2021-07-10T09:26:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 844309504 bytes 2021-07-10T09:26:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 899963904 bytes 2021-07-10T09:26:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 952263680 bytes 2021-07-10T09:26:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 976967680 bytes 2021-07-10T09:26:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1047110656 bytes 2021-07-10T09:26:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1128249344 bytes 2021-07-10T09:26:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1140471808 bytes 2021-07-10T09:26:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1142525952 bytes 2021-07-10T09:26:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1158425600 bytes 2021-07-10T09:26:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1218941952 bytes 2021-07-10T09:26:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1261906944 bytes 2021-07-10T09:26:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1291051008 bytes 2021-07-10T09:26:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:26:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:26:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:26:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:26:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:26:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:26:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:26:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:26:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:26:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:26:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:26:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:26:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:26:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:26:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:26:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:26:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:26:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:26:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:26:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":17152921.6,"bytes_per_snapshot":502802944.0,"local_snapshot_timestamp":1625909148,"remote_snapshot_timestamp":1625909148,"replay_state":"idle"} 2021-07-10T09:26:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.2 on smithi105 2021-07-10T09:26:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:26:01 2021-07-10T09:26:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:26:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:26:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:26:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:26:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:26:01 2021-07-10T09:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:26:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":17152921.6,"bytes_per_snapshot":502802944.0,"local_snapshot_timestamp":1625909148,"remote_snapshot_timestamp":1625909148,"replay_state":"idle"} 2021-07-10T09:26:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:26:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:26:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:26:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:26:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.2 on smithi105 2021-07-10T09:26:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:26:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:26:01' 2021-07-10T09:26:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:26:01' 2021-07-10T09:26:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:26:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:26:01' 2021-07-10T09:26:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:26:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:26:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:26:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:26:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:26:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:26:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:26:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:26:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:26:01' 2021-07-10T09:26:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:26:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:26:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:26:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:26:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:26:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap7 2021-07-10T09:26:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap7 2021-07-10T09:26:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:26:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap7 2021-07-10T09:26:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:26:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:26:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:26:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:26:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:26:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:26:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:26:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:26:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:26:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:26:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:26:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:26:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:26:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:26:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:26:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:26:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:26:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:26:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:26:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:26:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:26:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:26:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:26:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:26:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:26:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:26:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:26:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:26:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:26:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:26:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:26:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:26:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:26:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:26:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:26:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:26:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:26:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:26:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:26:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:26:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:26:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:26:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:26:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:26:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:26:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:26:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:26:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:26:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:26:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:26:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:26:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:26:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:26:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:26:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:26:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:26:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:26:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:26:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:26:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:26:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:26:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:26:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:26:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:26:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:26:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:26:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:26:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:26:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:26:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:26:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:26:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:26:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:26:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:26:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:26:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:26:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:26:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:26:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:26:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:26:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:26:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:26:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:26:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:26:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:26:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:26:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:26:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:26:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:26:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:26:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:26:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:26:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:26:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:26:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:26:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:26:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:26:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:26:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:26:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:26:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:26:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:26:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:26:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:26:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:26:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:26:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:26:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:26:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:26:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:26:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:26:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:26:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:26:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:26:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:26:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:26:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:26:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:26:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:26:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:26:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:26:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:26:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 30 2021-07-10T09:26:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:26:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:26:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:26:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:26:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=30 2021-07-10T09:26:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:26:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:26:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:26:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:26:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:26:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:26:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:26:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:26:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:26:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:26:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:26:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 30 = '' 2021-07-10T09:26:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:26:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:26:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:26:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:26:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:26:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:26:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:26:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:26:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:26:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:26:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 30 = '' 2021-07-10T09:26:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:26:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:26:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:26:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:26:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:26:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:26:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:26:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:26:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:26:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 30 = '' 2021-07-10T09:26:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:26:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:26:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:26:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:26:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:26:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:26:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:26:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:26:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:26:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:26:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 30 = '' 2021-07-10T09:26:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:26:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:26:40.896 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:26:40.895+0000 7f4236192700 -1 reset not still connected to 0x55a453a5add0 2021-07-10T09:26:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:26:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:26:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:26:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:26:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=30 2021-07-10T09:26:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 30 = 30 2021-07-10T09:26:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2021-07-10T09:26:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:26:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap7 2021-07-10T09:26:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:26:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:26:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:26:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2021-07-10T09:26:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:26:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:26:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:26:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 14 2021-07-10T09:26:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:26:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:26:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:26:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:26:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:26:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:26:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:26:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:25:03 2021 2021-07-10T09:26:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:26:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:26:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:26:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:26:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:26:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:26:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap6-clean ']' 2021-07-10T09:26:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap6-clean 2021-07-10T09:26:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap6-clean 2021-07-10T09:26:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:26:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:26:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:26:43.997 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:26:43.995+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:26:43.997 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:26:43.995+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:26:43.998 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:26:43.995+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:26:44.111 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:26:44.109+0000 7f511128e700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:26:44.111 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:26:44.109+0000 7f5114a95700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2021-07-10T09:26:44.421 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:26:44.419+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:26:44.539 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:26:44.535+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:26:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:44.718 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:44.718+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:26:44.719 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:44.718+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:26:44.719 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:44.718+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:26:44.795 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:44.794+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:26:44.868 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:44.867+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:26:44.868 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:44.868+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:26:44.871 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:44.869+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:26:44.941 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:26:44.940+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:26:45.092 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:26:45.091+0000 7f4226973700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:26:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:26:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap7 2021-07-10T09:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap7 2021-07-10T09:26:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:26:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:26:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:26:46.045 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:26:46.043+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:26:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:26:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:46.680 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:46.680+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:26:46.812 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:46.812+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:26:46.813 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:46.812+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:26:46.813 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:46.812+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:26:46.814 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:46.813+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:26:46.814 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:46.813+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:26:46.815 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:46.814+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:26:47.212 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:26:47.212+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:26:47.284 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:47.283+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:26:47.285 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:47.284+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:26:47.285 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:47.284+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:26:47.286 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:47.286+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:26:47.358 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:47.357+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:26:47.360 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:47.358+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:26:47.360 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:26:47.359+0000 7f3176865700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:26:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:26:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:26:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:26:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap7-clean 2021-07-10T09:26:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap7-clean 2021-07-10T09:26:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:26:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap7-clean 2021-07-10T09:26:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:26:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:26:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:26:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:26:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:26:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:26:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:26:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:26:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:26:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:26:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 42863616 bytes 2021-07-10T09:26:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 74987520 bytes 2021-07-10T09:26:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 114403328 bytes 2021-07-10T09:26:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 183394304 bytes 2021-07-10T09:26:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 209139712 bytes 2021-07-10T09:26:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 237705216 bytes 2021-07-10T09:26:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 289486848 bytes 2021-07-10T09:26:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 338825216 bytes 2021-07-10T09:26:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 390084608 bytes 2021-07-10T09:26:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 477342720 bytes 2021-07-10T09:26:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 508730368 bytes 2021-07-10T09:26:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 575535104 bytes 2021-07-10T09:26:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 603897856 bytes 2021-07-10T09:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 604509184 bytes 2021-07-10T09:26:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 657920000 bytes 2021-07-10T09:26:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 689281024 bytes 2021-07-10T09:26:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 751377408 bytes 2021-07-10T09:26:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 769176576 bytes 2021-07-10T09:26:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 823231488 bytes 2021-07-10T09:26:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 823839744 bytes 2021-07-10T09:26:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 855111680 bytes 2021-07-10T09:26:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 872270848 bytes 2021-07-10T09:26:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 933307392 bytes 2021-07-10T09:26:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 993358848 bytes 2021-07-10T09:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1032337408 bytes 2021-07-10T09:26:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1061563392 bytes 2021-07-10T09:26:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1139270656 bytes 2021-07-10T09:26:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1184967680 bytes 2021-07-10T09:26:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1210825728 bytes 2021-07-10T09:26:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1240577024 bytes 2021-07-10T09:26:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1258685440 bytes 2021-07-10T09:26:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1293214720 bytes 2021-07-10T09:26:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1303515136 bytes 2021-07-10T09:26:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1341346816 bytes 2021-07-10T09:26:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1435742208 bytes 2021-07-10T09:26:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1445000192 bytes 2021-07-10T09:26:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1533277184 bytes 2021-07-10T09:26:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1583242240 bytes 2021-07-10T09:26:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1606870016 bytes 2021-07-10T09:26:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1658980352 bytes 2021-07-10T09:26:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1698850816 bytes 2021-07-10T09:26:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1805374464 bytes 2021-07-10T09:26:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1917837312 bytes 2021-07-10T09:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1976931328 bytes 2021-07-10T09:27:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1984533504 bytes 2021-07-10T09:27:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 2014871552 bytes 2021-07-10T09:27:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:27:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:27:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:27:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:27:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:27:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:27:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:27:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:27:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:27:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:27:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:27:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:27:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:27:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:27:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:27:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:27:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:27:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":16275524.27,"bytes_per_snapshot":501426688.0,"local_snapshot_timestamp":1625909197,"remote_snapshot_timestamp":1625909197,"replay_state":"idle"} 2021-07-10T09:27:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.2 on smithi105 2021-07-10T09:27:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:27:01 2021-07-10T09:27:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:27:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:27:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:27:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:27:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:27:01 2021-07-10T09:27:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:27:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":16275524.27,"bytes_per_snapshot":501426688.0,"local_snapshot_timestamp":1625909197,"remote_snapshot_timestamp":1625909197,"replay_state":"idle"} 2021-07-10T09:27:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:27:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:27:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:27:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:27:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.2 on smithi105 2021-07-10T09:27:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:27:01' 2021-07-10T09:27:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:27:01' 2021-07-10T09:27:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:27:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:27:01' 2021-07-10T09:27:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:27:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:27:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:27:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:27:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:27:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:27:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:27:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:27:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:27:01' 2021-07-10T09:27:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:27:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:27:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:27:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:27:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:27:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap8 2021-07-10T09:27:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap8 2021-07-10T09:27:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap8 2021-07-10T09:27:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:27:04.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.31792473617224826) > weight (0.25) 2021-07-10T09:27:04.494 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2021-07-10T09:27:04.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2021-07-10T09:27:04.495 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2021-07-10T09:27:04.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi105.stderr:2021-07-10T09:27:04.494+0000 7ff85daae700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36799) UID: 1000 2021-07-10T09:27:04.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:27:12.496 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2021-07-10T09:27:12.497 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:27:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:27:12.498 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2021-07-10T09:27:12.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2021-07-10T09:27:12.499 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2021-07-10T09:27:12.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2021-07-10T09:27:12.505 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2021-07-10T09:27:12.691 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:12.689+0000 7f69f674e700 -1 reset not still connected to 0x5630753f3380 2021-07-10T09:27:12.691 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:12.689+0000 7f69f674e700 -1 reset not still connected to 0x56307bd56c30 2021-07-10T09:27:12.692 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:12.689+0000 7f69f674e700 -1 reset not still connected to 0x56307bd57790 2021-07-10T09:27:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:27:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:27:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:27:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:27:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:27:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:27:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:27:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:27:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:27:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:27:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:27:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:27:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:27:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:27:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:27:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:27:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:27:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:27:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:27:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:27:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:27:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:27:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:27:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:27:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:27:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:27:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:27:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:27:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:27:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:27:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:27:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:27:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:27:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:27:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:27:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:27:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:27:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:27:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:27:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:27:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:27:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:27:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:27:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:27:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:27:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:27:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:27:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:27:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:27:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:27:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:27:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:27:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:27:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:27:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:27:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:27:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:27:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:27:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:27:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:27:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:27:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:27:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:27:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:27:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:27:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:27:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:27:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:27:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:27:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:27:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:27:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:27:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:27:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:27:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:27:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:27:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:27:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:27:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:27:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:27:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:27:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:27:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:27:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:27:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:27:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:27:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:27:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:27:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:27:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:27:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:27:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:27:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:27:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:27:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:27:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:27:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:27:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:27:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:27:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:27:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:27:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:27:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:27:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:27:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:27:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:27:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:27:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:27:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:27:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:27:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:27:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:27:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:27:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:27:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:27:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:27:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:27:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:27:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:27:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:27:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:27:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:27:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:27:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:27:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:27:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:27:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:27:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:27:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:27:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:27:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:27:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:27:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:27:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:27:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:27:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:27:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:27:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:27:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:27:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:27:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:27:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:27:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:27:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:27:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:27:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:27:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:27:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:27:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:27:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:27:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:27:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 34 2021-07-10T09:27:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:27:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:27:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:27:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:27:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:27:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=34 2021-07-10T09:27:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:27:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:27:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:27:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:27:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:27:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:27:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:27:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:27:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:27:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:27:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:27:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 34 = '' 2021-07-10T09:27:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:27:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:27:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:27:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:27:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:27:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:27:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:27:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:27:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:27:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:27:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 34 = '' 2021-07-10T09:27:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:27:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:27:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:27:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:27:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:27:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:27:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:27:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:27:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 34 = '' 2021-07-10T09:27:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:27:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:27:35.513 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4259781381437333) > weight (0.25) 2021-07-10T09:27:35.513 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9790104067163471) > weight (0.25) 2021-07-10T09:27:35.513 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9516389478850329) > weight (0.25) 2021-07-10T09:27:35.514 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5590301916421593) > weight (0.25) 2021-07-10T09:27:35.514 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2021-07-10T09:27:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:27:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:27:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:27:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:27:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:27:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:27:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:27:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 34 = '' 2021-07-10T09:27:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:27:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:27:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:27:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:27:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:27:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:27:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=34 2021-07-10T09:27:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 34 = 34 2021-07-10T09:27:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2021-07-10T09:27:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap8 2021-07-10T09:27:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:27:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:27:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2021-07-10T09:27:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:27:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:27:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:27:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 16 2021-07-10T09:27:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:27:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:27:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:27:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:27:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:27:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:27:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:27:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:26:44 2021 2021-07-10T09:27:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:27:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:27:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:27:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:27:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:27:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:27:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap7-clean ']' 2021-07-10T09:27:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap7-clean 2021-07-10T09:27:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap7-clean 2021-07-10T09:27:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:27:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:27:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:27:38.993 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:27:38.991+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:27:38.993 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:27:38.991+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:27:39.104 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:27:39.102+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:27:39.274 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:39.273+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:27:39.377 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:39.375+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:39.377 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:39.375+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:39.488 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:39.486+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:39.489 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:39.486+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:39.538 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:27:39.536+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:27:39.539 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:27:39.537+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:27:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:27:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:39.822 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:39.821+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:27:39.823 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:39.821+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:27:39.823 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:39.821+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:27:39.899 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:39.894+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:27:39.973 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:39.968+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:27:39.973 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:39.969+0000 7f4228977700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:27:40.182 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:40.177+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:27:40.182 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:40.177+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:27:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:27:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap8 2021-07-10T09:27:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap8 2021-07-10T09:27:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:27:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:27:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:27:41.251 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:41.249+0000 7f69e9734700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:41.252 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:41.250+0000 7f69e672e700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:41.252 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:41.250+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:41.252 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:41.250+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:27:41.434 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:27:41.431+0000 7f69e572c700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:27:41.502 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:27:41.501+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:27:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:27:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:41.674 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:41.673+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:27:41.676 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:41.675+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:27:41.786 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:41.784+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:27:41.787 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:41.784+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:27:41.787 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:41.784+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:27:41.854 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:41.853+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:27:41.855 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:27:41.853+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:27:41.941 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:41.939+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:27:41.942 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:41.939+0000 7f422496f700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2021-07-10T09:27:42.100 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:42.098+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:27:42.101 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:42.098+0000 7f4226973700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:27:42.101 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:42.098+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2021-07-10T09:27:42.102 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:27:42.099+0000 7f4225170700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:27:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:27:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:27:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap8-clean 2021-07-10T09:27:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap8-clean 2021-07-10T09:27:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap8-clean 2021-07-10T09:27:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:27:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:27:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:27:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:27:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=13 2021-07-10T09:27:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:27:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:27:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 22260736 bytes 2021-07-10T09:27:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 50630656 bytes 2021-07-10T09:27:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 80710656 bytes 2021-07-10T09:27:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 103951360 bytes 2021-07-10T09:27:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 158107648 bytes 2021-07-10T09:27:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 181343232 bytes 2021-07-10T09:27:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 268610560 bytes 2021-07-10T09:27:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 295991296 bytes 2021-07-10T09:27:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 368242688 bytes 2021-07-10T09:27:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 410132480 bytes 2021-07-10T09:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 463990784 bytes 2021-07-10T09:27:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 527471616 bytes 2021-07-10T09:27:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 620236800 bytes 2021-07-10T09:27:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 676840448 bytes 2021-07-10T09:27:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 729894912 bytes 2021-07-10T09:27:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 767248384 bytes 2021-07-10T09:27:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 794487808 bytes 2021-07-10T09:27:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 831125504 bytes 2021-07-10T09:27:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 900081664 bytes 2021-07-10T09:27:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 900795392 bytes 2021-07-10T09:27:49.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 920664064 bytes 2021-07-10T09:27:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 980774912 bytes 2021-07-10T09:27:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1033810944 bytes 2021-07-10T09:27:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1049851904 bytes 2021-07-10T09:27:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1074192384 bytes 2021-07-10T09:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1074794496 bytes 2021-07-10T09:27:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1138867200 bytes 2021-07-10T09:27:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1186395136 bytes 2021-07-10T09:27:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1212648448 bytes 2021-07-10T09:27:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1275265024 bytes 2021-07-10T09:27:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1298453504 bytes 2021-07-10T09:27:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1351664640 bytes 2021-07-10T09:27:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1355186176 bytes 2021-07-10T09:27:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1398835200 bytes 2021-07-10T09:27:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1450245120 bytes 2021-07-10T09:27:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1496260608 bytes 2021-07-10T09:27:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1525645312 bytes 2021-07-10T09:27:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1602282496 bytes 2021-07-10T09:27:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1640921088 bytes 2021-07-10T09:27:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1704737792 bytes 2021-07-10T09:27:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1747915776 bytes 2021-07-10T09:27:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1793081344 bytes 2021-07-10T09:27:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1850716160 bytes 2021-07-10T09:27:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1899594752 bytes 2021-07-10T09:27:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1908028416 bytes 2021-07-10T09:27:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 1933565952 bytes 2021-07-10T09:27:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 2004043776 bytes 2021-07-10T09:27:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4800 IOs, 2004670464 bytes 2021-07-10T09:27:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4900 IOs, 2029800448 bytes 2021-07-10T09:27:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:27:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:27:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:27:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:27:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:27:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:27:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:27:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:27:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:27:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:27:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:27:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:27:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:27:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:27:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:27:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:27:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:27:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:27:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:27:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":501426688.0,"local_snapshot_timestamp":1625909197,"remote_snapshot_timestamp":1625909197,"replay_state":"idle"} 2021-07-10T09:27:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.2 on smithi105 2021-07-10T09:27:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:27:31 2021-07-10T09:27:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:27:31 2021-07-10T09:27:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:27:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":501426688.0,"local_snapshot_timestamp":1625909197,"remote_snapshot_timestamp":1625909197,"replay_state":"idle"} 2021-07-10T09:27:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:27:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:27:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:27:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:27:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.2 on smithi105 2021-07-10T09:27:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:27:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:27:31' 2021-07-10T09:27:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:27:31' 2021-07-10T09:27:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:27:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:27:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:27:31' 2021-07-10T09:27:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:27:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:27:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:27:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:27:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:27:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:27:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:27:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:27:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:27:31' 2021-07-10T09:27:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:27:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:27:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:27:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:27:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:27:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:27:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap9 2021-07-10T09:27:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap9 2021-07-10T09:27:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:27:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:27:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:27:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap9 2021-07-10T09:27:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:28:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:28:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:28:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:28:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:28:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:28:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:28:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:28:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:28:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:28:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:28:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:28:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:28:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:28:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:28:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:28:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:28:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:28:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:28:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:28:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:28:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:28:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:28:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:28:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:28:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:28:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:28:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:28:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:28:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:28:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:28:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:28:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:28:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:28:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:28:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:28:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:28:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:28:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:28:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:28:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:28:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:28:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:28:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:28:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:28:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:28:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:28:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:28:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:28:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:28:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:28:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:28:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:28:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:28:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:28:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:28:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:28:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:28:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:28:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:28:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:28:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:28:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:28:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:28:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:28:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:28:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:28:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:28:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:28:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:28:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:28:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:28:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:28:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:28:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:28:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:28:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:28:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:28:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:28:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:28:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:28:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:28:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:28:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:28:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:28:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:28:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:28:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:28:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:28:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:28:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:28:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:28:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:28:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:28:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:28:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:28:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:28:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:28:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:28:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:28:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:28:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:28:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:28:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:28:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:28:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:28:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:28:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:28:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:28:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:28:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:28:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:28:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:28:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:28:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:28:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:28:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:28:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:28:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:28:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:28:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:28:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:28:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:28:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:28:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:28:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:28:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:28:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:28:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:28:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:28:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:28:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:28:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:28:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:28:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:28:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:28:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:28:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:28:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:28:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:28:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:28:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:28:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:28:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 38 2021-07-10T09:28:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:28:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:28:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:28:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:28:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:28:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:28:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=38 2021-07-10T09:28:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:28:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:28:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:28:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:28:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:28:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:28:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:28:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:28:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:28:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:28:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:28:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 38 = '' 2021-07-10T09:28:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:28:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:28:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:28:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:28:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:28:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:28:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 38 = '' 2021-07-10T09:28:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:28:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:28:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:28:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:28:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:28:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:28:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:28:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:28:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:28:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:28:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 38 = '' 2021-07-10T09:28:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:28:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:28:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:28:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:28:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:28:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:28:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:28:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:28:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:28:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:28:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 38 = '' 2021-07-10T09:28:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:28:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:28:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:28:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:28:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:28:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:28:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=38 2021-07-10T09:28:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 38 = 38 2021-07-10T09:28:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:28:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2021-07-10T09:28:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap9 2021-07-10T09:28:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:28:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:28:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:28:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2021-07-10T09:28:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 18 2021-07-10T09:28:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:28:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:28:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:28:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:28:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:28:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:28:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:28:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:26:44 2021 2021-07-10T09:28:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:28:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:28:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:28:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:28:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:28:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:28:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:28:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap8-clean ']' 2021-07-10T09:28:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap8-clean 2021-07-10T09:28:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap8-clean 2021-07-10T09:28:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:28:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:28:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:28:35.056 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:35.054+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:28:35.394 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:35.392+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:28:35.495 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:28:35.492+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:28:35.495 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:28:35.493+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:28:35.549 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:35.547+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:28:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:28:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:35.853 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:35.850+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:28:35.853 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:35.851+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:28:35.917 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:35.915+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:28:35.918 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:35.916+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:28:35.918 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:35.917+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:28:36.140 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:36.139+0000 7f4229178700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:28:36.141 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:36.139+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:28:36.142 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:36.141+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:28:36.280 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:36.278+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:28:36.281 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:36.279+0000 7f4226973700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:28:36.281 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:36.280+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:28:36.449 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:36.448+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:28:36.450 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:36.448+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:28:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:28:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:28:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap9 2021-07-10T09:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap9 2021-07-10T09:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:28:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:28:37.021 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:37.019+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:28:37.021 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:37.019+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:28:37.129 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:37.127+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:28:37.130 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:37.127+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:28:37.130 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:37.128+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:28:37.289 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:28:37.288+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:28:37.290 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:28:37.289+0000 7f69e472a700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2021-07-10T09:28:37.515 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:28:37.514+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:28:37.518 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:28:37.514+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:28:37.569 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:37.567+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:28:37.570 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:28:37.568+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:28:37.626 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:28:37.624+0000 7f69e7f31700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:28:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:28:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:37.807 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.805+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:28:37.807 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.805+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:28:37.807 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.805+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:28:37.808 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.806+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:28:37.878 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.877+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:28:37.879 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.877+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:28:37.947 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.946+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:28:37.948 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:37.946+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:28:38.024 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:38.022+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:28:38.027 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:38.025+0000 7f422496f700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:28:38.178 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:28:38.177+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:28:38.247 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:38.245+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:28:38.396 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:38.386+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:28:38.396 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:38.387+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:28:38.459 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:28:38.458+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:28:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:28:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:28:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:28:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap9-clean 2021-07-10T09:28:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap9-clean 2021-07-10T09:28:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:28:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap9-clean 2021-07-10T09:28:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:28:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:28:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror test 2021-07-10T09:28:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:28:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:28:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:28:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:28:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2021-07-10T09:28:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 41330688 bytes 2021-07-10T09:28:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 54201344 bytes 2021-07-10T09:28:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 120660992 bytes 2021-07-10T09:28:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 175238144 bytes 2021-07-10T09:28:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 208334848 bytes 2021-07-10T09:28:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 243476480 bytes 2021-07-10T09:28:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 253190144 bytes 2021-07-10T09:28:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 300266496 bytes 2021-07-10T09:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 349689856 bytes 2021-07-10T09:28:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 399380480 bytes 2021-07-10T09:28:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 417805312 bytes 2021-07-10T09:28:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 418979840 bytes 2021-07-10T09:28:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 439914496 bytes 2021-07-10T09:28:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 549641216 bytes 2021-07-10T09:28:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 569303040 bytes 2021-07-10T09:28:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 601057280 bytes 2021-07-10T09:28:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 625683456 bytes 2021-07-10T09:28:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 664197120 bytes 2021-07-10T09:28:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 679395328 bytes 2021-07-10T09:28:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 712373248 bytes 2021-07-10T09:28:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 776973312 bytes 2021-07-10T09:28:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 809468928 bytes 2021-07-10T09:28:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 834610176 bytes 2021-07-10T09:28:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 857951232 bytes 2021-07-10T09:28:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 916291584 bytes 2021-07-10T09:28:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 942870528 bytes 2021-07-10T09:28:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 943464448 bytes 2021-07-10T09:28:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1028977664 bytes 2021-07-10T09:28:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1117066240 bytes 2021-07-10T09:28:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1120041984 bytes 2021-07-10T09:28:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1161964544 bytes 2021-07-10T09:28:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1185098752 bytes 2021-07-10T09:28:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1201489920 bytes 2021-07-10T09:28:48.515 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6203916829570982) > weight (0.25) 2021-07-10T09:28:48.516 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8998408598965844) > weight (0.25) 2021-07-10T09:28:48.516 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2021-07-10T09:28:48.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2021-07-10T09:28:48.517 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2021-07-10T09:28:48.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:2021-07-10T09:28:48.519+0000 7fd8624cd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36684) UID: 1000 2021-07-10T09:28:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1243819008 bytes 2021-07-10T09:28:48.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:28:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1253800960 bytes 2021-07-10T09:28:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1302309888 bytes 2021-07-10T09:28:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1329399808 bytes 2021-07-10T09:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-07-10T09:28:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:28:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:28:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:28:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:28:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:28:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:28:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:28:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:28:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:28:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:28:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:28:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:28:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:28:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:28:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+stopped 2021-07-10T09:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: stopped 2021-07-10T09:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:28:48 2021-07-10T09:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:28:31 2021-07-10T09:28:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:28:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:28:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:28:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:28:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:28:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:28:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:28:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:28:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:28:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:28:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:28:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:28:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:28:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:28:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+stopped 2021-07-10T09:28:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: stopped 2021-07-10T09:28:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:28:48 2021-07-10T09:28:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:28:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:28:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+stopped 2021-07-10T09:28:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: local image is primary 2021-07-10T09:28:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:28:31 2021-07-10T09:28:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:28:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:28:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:28:53.505 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2021-07-10T09:28:53.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2021-07-10T09:28:53.506 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2021-07-10T09:28:53.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:28:53.506+0000 7f0558aa0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36764) UID: 1000 2021-07-10T09:28:53.518 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2021-07-10T09:28:53.518 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:28:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:28:53.519 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2021-07-10T09:28:53.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2021-07-10T09:28:53.520 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2021-07-10T09:28:53.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2021-07-10T09:28:53.525 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2021-07-10T09:28:53.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:28:55.508 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2021-07-10T09:28:55.508 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:28:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:28:55.509 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2021-07-10T09:28:55.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:28:55.510 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:28:55.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:28:55.513 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2021-07-10T09:28:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:28:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:28:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:28:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:28:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:28:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:28:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:28:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:28:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:28:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:28:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:28:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:28:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:28:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:28:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:28:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:28:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:28:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:28:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:28:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:28:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:29:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:29:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:29:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:29:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:29:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:29:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:29:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:29:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:29:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:29:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:29:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:29:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:29:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:29:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:29:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:29:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:29:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:29:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:29:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:29:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:29:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:29:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:29:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:29:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:29:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:29:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:29:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:29:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:29:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:29:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:29:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:29:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:29:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:29:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:29:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:29:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:29:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:29:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:29:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:29:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 1 2021-07-10T09:29:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:29:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:29:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2021-07-10T09:29:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:29:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local description_pattern= 2021-07-10T09:29:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local service_pattern= 2021-07-10T09:29:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ CEPH_ARGS= 2021-07-10T09:29:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2021-07-10T09:29:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:test: 2021-07-10T09:29:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: global_id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:29:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: up+replaying 2021-07-10T09:29:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1625909308,"remote_snapshot_timestamp":1625909308,"replay_state":"idle"} 2021-07-10T09:29:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: service: mirror.3 on smithi105 2021-07-10T09:29:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10 09:29:17 2021-07-10T09:29:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: peer_sites: 2021-07-10T09:29:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: name: cluster2 2021-07-10T09:29:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: state: down+unknown 2021-07-10T09:29:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: description: status not found 2021-07-10T09:29:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: last_update: 2021-07-10T09:29:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: state: up+replaying 2021-07-10T09:29:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^ description: .*' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1625909308,"remote_snapshot_timestamp":1625909308,"replay_state":"idle"} 2021-07-10T09:29:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:29:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:29:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep '^up+' 2021-07-10T09:29:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:29:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: service: mirror.3 on smithi105 2021-07-10T09:29:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.mirror_status 2021-07-10T09:29:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'last_update=2021-07-10 09:29:17' 2021-07-10T09:29:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2021-07-10 09:29:17' 2021-07-10T09:29:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state_pattern=up+replaying 2021-07-10T09:29:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local 'prev_last_update=2021-07-10 09:29:17' 2021-07-10T09:29:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror.mirror_status 2021-07-10T09:29:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror.mirror_status 2021-07-10T09:29:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror.mirror_status 2021-07-10T09:29:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-07-10T09:29:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:29:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_update state 2021-07-10T09:29:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2021-07-10T09:29:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_update='2021-07-10 09:29:17' 2021-07-10T09:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2021-07-10T09:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=up+replaying 2021-07-10T09:29:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo up+replaying 2021-07-10T09:29:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep up+replaying 2021-07-10T09:29:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:up+replaying 2021-07-10T09:29:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap10 2021-07-10T09:29:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap10 2021-07-10T09:29:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap10 2021-07-10T09:29:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:29:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:29:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:29:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:29:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:29:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:29:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:29:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:29:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:29:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:29:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:29:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:29:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:29:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:29:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:29:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:29:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:29:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:29:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:29:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:29:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:29:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:29:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:29:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:29:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:29:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:29:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:29:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:29:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:29:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:29:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:29:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:29:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:29:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:29:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:29:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:29:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:29:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:29:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:29:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:29:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:29:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:29:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:29:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:29:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:29:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:29:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:29:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:29:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:29:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:29:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:29:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:29:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:29:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:29:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:29:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:29:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:29:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:29:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:29:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:29:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:29:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:29:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:29:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:29:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:29:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:29:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:29:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:29:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:29:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:29:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:29:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:29:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:29:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:29:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:29:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:29:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:29:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:29:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:29:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:29:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:29:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:29:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:29:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:29:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:29:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:29:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:29:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:29:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:29:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:29:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:29:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:29:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:29:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:29:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:29:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:29:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:29:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:29:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:29:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:29:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:29:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:29:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:29:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:29:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:29:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:29:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:29:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:29:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:29:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:29:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:29:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:29:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:29:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:29:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:29:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:29:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:29:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:29:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:29:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:29:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:29:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:29:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:29:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:29:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 42 2021-07-10T09:29:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:29:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:29:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:29:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:29:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=42 2021-07-10T09:29:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:29:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:29:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:29:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:29:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:29:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 42 = '' 2021-07-10T09:29:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:29:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:29:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:29:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:29:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:29:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 42 = '' 2021-07-10T09:29:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:29:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:29:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:29:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:29:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:29:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 42 = '' 2021-07-10T09:29:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:29:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:29:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:29:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:29:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:29:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 42 = '' 2021-07-10T09:29:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:29:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=42 2021-07-10T09:29:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 42 = 42 2021-07-10T09:29:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2021-07-10T09:29:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap10 2021-07-10T09:29:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:29:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:29:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:29:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2021-07-10T09:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 20 2021-07-10T09:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:29:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:29:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:29:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:29:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:29:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:28:35 2021 2021-07-10T09:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:29:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:29:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:29:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snap9-clean ']' 2021-07-10T09:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap9-clean 2021-07-10T09:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap9-clean 2021-07-10T09:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:30.251 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:29:30.249+0000 7f5115a97700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:29:30.251 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:29:30.249+0000 7f5116a99700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:29:30.421 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:30.419+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:29:30.481 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:30.479+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:30.702 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:29:30.700+0000 7f72ca097700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:29:30.817 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:30.814+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:30.820 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:30.814+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:30.820 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:30.815+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:30.820 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:30.815+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:31.040 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:31.036+0000 7f4225170700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:31.346 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:31.345+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap10 2021-07-10T09:29:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap10 2021-07-10T09:29:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:31.942 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:31.941+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:32.212 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:32.210+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:32.511 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:32.508+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:32.511 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:32.509+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:29:32.512 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:32.510+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2021-07-10T09:29:32.516 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:32.513+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2021-07-10T09:29:32.516 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:32.513+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2021-07-10T09:29:32.640 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:32.638+0000 7f69f674e700 -1 reset not still connected to 0x56305ec611e0 2021-07-10T09:29:32.641 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:32.638+0000 7f69f674e700 -1 reset not still connected to 0x56306add3450 2021-07-10T09:29:32.641 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:32.638+0000 7f69f674e700 -1 reset not still connected to 0x5630753f3380 2021-07-10T09:29:32.641 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:32.638+0000 7f69f674e700 -1 reset not still connected to 0x563077e0cb60 2021-07-10T09:29:32.641 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:32.638+0000 7f69f674e700 -1 reset not still connected to 0x56307bd56c30 2021-07-10T09:29:32.642 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:32.638+0000 7f69f674e700 -1 reset not still connected to 0x56307bd57790 2021-07-10T09:29:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:32.710 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:32.708+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:32.815 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:32.813+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:32.818 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:32.814+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:32.819 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:32.815+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ clean_snap_name=snap10-clean 2021-07-10T09:29:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror test snap10-clean 2021-07-10T09:29:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap10-clean 2021-07-10T09:29:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2021-07-10T09:29:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-07-10T09:29:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-07-10T09:29:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:29:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:29:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:29:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:29:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror test started 2021-07-10T09:29:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:29:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:29:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:29:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2021-07-10T09:29:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:29:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:29:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:29:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:29:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:29:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:29:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:29:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:29:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:29:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-07-10T09:29:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:29:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:29:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:29:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:29:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:29:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:29:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:29:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:29:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:29:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:29:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:29:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:29:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:29:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:29:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:29:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:29:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:29:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:29:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:29:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-07-10T09:29:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:29:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:29:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:29:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-07-10T09:29:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:29:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:29:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:29:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:29:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:29:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:29:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:29:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:29:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:29:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:29:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:29:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:29:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:29:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-07-10T09:29:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:29:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:29:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:29:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-07-10T09:29:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:29:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:29:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:29:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:29:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:29:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:29:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:29:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:29:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:29:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:29:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:29:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:29:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:29:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:29:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:29:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:29:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:29:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:29:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:29:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-07-10T09:29:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-07-10T09:29:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:29:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:29:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:29:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-07-10T09:29:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:29:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:29:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:29:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:29:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:29:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:29:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:29:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:29:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:29:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:29:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:29:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:29:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:29:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:29:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:29:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:29:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:29:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:29:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:29:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:29:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:29:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:29:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:29:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:29:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:29:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:29:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:29:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:29:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:29:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:29:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:29:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:29:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-07-10T09:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:29:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:29:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:29:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:29:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:29:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:29:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:29:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-07-10T09:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:29:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:29:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:29:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2021-07-10T09:29:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:29:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-test.status 2021-07-10T09:29:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-test.status 2021-07-10T09:29:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:29:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-test.status 2021-07-10T09:29:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-test.status 2021-07-10T09:29:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:29:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror test 2021-07-10T09:29:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2021-07-10T09:29:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 45 2021-07-10T09:29:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:29:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:29:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.status 2021-07-10T09:29:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2021-07-10T09:29:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:29:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=45 2021-07-10T09:29:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:29:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:29:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:29:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:29:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:29:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 45 = '' 2021-07-10T09:29:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:29:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:29:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:29:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:29:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:29:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:29:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 45 = '' 2021-07-10T09:29:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:29:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:29:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.status 2021-07-10T09:29:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2021-07-10T09:29:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:29:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=45 2021-07-10T09:29:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 45 = 45 2021-07-10T09:29:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2021-07-10T09:29:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:29:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap10-clean 2021-07-10T09:29:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:29:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:29:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2021-07-10T09:29:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'test': 2021-07-10T09:29:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 512 MiB in 128 objects 2021-07-10T09:29:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:29:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 21 2021-07-10T09:29:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10a95c32b3f8 2021-07-10T09:29:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10a95c32b3f8 2021-07-10T09:29:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:29:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:29:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:29:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:29:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:29:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:28:35 2021 2021-07-10T09:29:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:20:33 2021 2021-07-10T09:29:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:29:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:29:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:29:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: da2d3b5a-2ee5-4d79-a8ca-4914b71eb9d9 2021-07-10T09:29:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:29:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 10 2021-07-10T09:29:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:29:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap1 2021-07-10T09:29:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap1 2021-07-10T09:29:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap1 2021-07-10T09:29:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:39.279 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.277+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:39.280 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.277+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:39.280 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.278+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:39.280 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.278+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:39.334 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.333+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:39.448 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.446+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:39.507 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.505+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:39.507 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.505+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:29:39.558 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:29:39.556+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:29:39.558 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:29:39.556+0000 7f5111a8f700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:29:39.755 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:39.753+0000 7f69e7f31700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:39.821 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.820+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:39.822 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.820+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:39.877 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:39.875+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:40.007 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:40.000+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:29:40.007 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:40.002+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:29:40.086 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:40.084+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:40.148 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:40.145+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:40.149 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:40.145+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:40.217 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.216+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:40.219 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.216+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:40.219 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.217+0000 7f4228977700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:29:40.219 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.217+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:29:40.220 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.218+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:40.363 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.362+0000 7f4229178700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:40.364 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.363+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:29:40.508 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.505+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:40.508 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.505+0000 7f4225170700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:40.508 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.506+0000 7f422a97b700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:40.509 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.506+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:40.510 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:40.508+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap1-clean 2021-07-10T09:29:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap1-clean 2021-07-10T09:29:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap1-clean 2021-07-10T09:29:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:41.375 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.374+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:41.376 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.374+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:41.485 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:29:41.484+0000 7f5112a91700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:29:41.486 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:29:41.485+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:29:41.627 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.625+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:41.627 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.626+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:41.671 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:41.670+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:29:41.673 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.671+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:29:41.673 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.671+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:29:41.724 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.723+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:41.726 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:41.724+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:42.090 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:42.088+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:42.090 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:42.089+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:42.167 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:42.166+0000 7f4228977700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:42.168 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:42.167+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:29:42.513 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:42.511+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:42.513 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:42.511+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:29:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap2 2021-07-10T09:29:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap2 2021-07-10T09:29:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap2 2021-07-10T09:29:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:43.570 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:43.568+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:43.629 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:43.626+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:43.629 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:43.627+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:29:43.629 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:43.627+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:29:43.630 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:43.628+0000 7f69e9734700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2021-07-10T09:29:43.682 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:43.681+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:43.683 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:43.681+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:43.843 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:43.842+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:43.941 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:43.939+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:43.941 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:43.939+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:43.942 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:43.940+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:43.942 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:43.940+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:29:43.942 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:43.940+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:44.006 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.005+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:29:44.006 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.005+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:29:44.007 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.005+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:44.154 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:44.153+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:44.302 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.300+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:44.303 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.301+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:44.449 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.448+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:44.451 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.448+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:44.458 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:44.450+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 9 2021-07-10T09:29:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap2-clean 2021-07-10T09:29:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap2-clean 2021-07-10T09:29:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap2-clean 2021-07-10T09:29:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:45.042 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.041+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:45.043 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.042+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:29:45.044 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.042+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:29:45.500 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.498+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:45.559 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:45.557+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2021-07-10T09:29:45.616 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.614+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:45.616 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.614+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:45.616 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.614+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:45.617 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:45.614+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:45.769 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:45.763+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:45.795 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:45.792+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:45.795 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:45.792+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:45.864 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:45.862+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:45.932 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:45.929+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 11 2021-07-10T09:29:45.932 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:45.930+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 8 2021-07-10T09:29:46.001 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:45.999+0000 7f422496f700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:46.001 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:46.000+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:29:46.003 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:46.001+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2021-07-10T09:29:46.004 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:46.002+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 9 2021-07-10T09:29:46.150 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:46.147+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:46.150 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:46.148+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:46.312 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:46.308+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:29:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap3 2021-07-10T09:29:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap3 2021-07-10T09:29:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap3 2021-07-10T09:29:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:46.968 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:46.967+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:47.131 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:47.130+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:47.293 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:47.293+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:47.294 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:47.293+0000 7f69e9734700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:47.358 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:47.358+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:47.359 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:47.358+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:48.172 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:48.170+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:48.173 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:48.170+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:48.173 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:48.171+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:48.462 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:48.460+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:29:48.462 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:48.460+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap3-clean 2021-07-10T09:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap3-clean 2021-07-10T09:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap3-clean 2021-07-10T09:29:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:49.018 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:49.014+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:29:49.018 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:49.014+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:49.281 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:49.281+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:49.336 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:49.336+0000 7f69e8f33700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:29:49.396 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:49.394+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:29:49.451 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:49.450+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:49.862 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:49.861+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:49.867 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:49.864+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:50.199 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:50.198+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:50.199 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:50.198+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:50.200 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:50.199+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:29:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:29:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap4 2021-07-10T09:29:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap4 2021-07-10T09:29:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap4 2021-07-10T09:29:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:51.397 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:51.396+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:51.398 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:51.398+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:29:51.530 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:51.529+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:51.531 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:51.529+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap4-clean 2021-07-10T09:29:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap4-clean 2021-07-10T09:29:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap4-clean 2021-07-10T09:29:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:52.458 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:52.457+0000 7f25794d0700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:52.462 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:52.457+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:52.991 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:52.990+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:52.992 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:52.992+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:53.268 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:53.267+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:29:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap5 2021-07-10T09:29:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap5 2021-07-10T09:29:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap5 2021-07-10T09:29:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:54.130 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:54.129+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:54.767 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:54.766+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:54.767 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:54.766+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:29:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:55.510 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:55.509+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:29:55.510 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:55.509+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:29:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap5-clean 2021-07-10T09:29:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap5-clean 2021-07-10T09:29:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap5-clean 2021-07-10T09:29:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:56.082 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:56.080+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:56.354 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:56.353+0000 7f69e572c700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:29:56.354 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:56.353+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:29:56.472 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:56.471+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:56.551 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:56.550+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:29:56.552 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:29:56.551+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:29:56.605 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:56.605+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:56.606 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:56.605+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:56.761 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:29:56.759+0000 7f72ca898700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:29:56.761 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:29:56.761+0000 7f72ca097700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:29:56.766 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:56.764+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:29:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap6 2021-07-10T09:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap6 2021-07-10T09:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap6 2021-07-10T09:29:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:58.245 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:58.244+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:58.245 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:58.245+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:58.306 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:58.305+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:29:58.311 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:58.307+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:29:58.311 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:29:58.308+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:29:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:58.505 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:58.505+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:58.506 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:58.506+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:58.506 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:58.506+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:29:58.760 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:58.759+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:29:59.013 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:59.013+0000 7f422a97b700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:29:59.132 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:59.013+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2021-07-10T09:29:59.132 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:59.014+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 7 2021-07-10T09:29:59.132 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:29:59.081+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:29:59.151 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:29:59.151+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:29:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:29:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap6-clean 2021-07-10T09:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap6-clean 2021-07-10T09:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap6-clean 2021-07-10T09:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:00.128 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:00.127+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:00.514 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6577325252732834) > weight (0.25) 2021-07-10T09:30:00.515 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2021-07-10T09:30:00.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2021-07-10T09:30:00.515 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2021-07-10T09:30:00.516 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2021-07-10T09:30:00.516 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:30:00.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi105.stderr:2021-07-10T09:30:00.514+0000 7f9fc0d29700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 43938) UID: 1000 2021-07-10T09:30:00.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:30:00.780 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:00.778+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:01.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap7 2021-07-10T09:30:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap7 2021-07-10T09:30:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap7 2021-07-10T09:30:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:01.617 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:01.614+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:30:01.617 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:01.614+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:30:01.618 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:01.615+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:30:01.731 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:01.729+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:01.902 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:01.900+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:30:02.014 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:02.012+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:02.392 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:02.390+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:02.393 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:02.390+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:02.393 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:02.390+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:02.393 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:02.391+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:30:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:02.541 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.539+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:02.621 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.619+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:02.622 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.620+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:30:02.622 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.621+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:02.692 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.691+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:02.693 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.692+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:30:02.766 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:02.764+0000 7f422a17a700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:30:02.767 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:02.765+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:30:02.767 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:02.765+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:30:02.953 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.952+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:02.954 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:02.952+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap7-clean 2021-07-10T09:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap7-clean 2021-07-10T09:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap7-clean 2021-07-10T09:30:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:03.870 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:03.868+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:03.982 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:30:03.980+0000 7f5112290700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 10 2021-07-10T09:30:03.982 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:30:03.980+0000 7f5113a93700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 10 2021-07-10T09:30:03.983 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:30:03.981+0000 7f5112a91700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 6 2021-07-10T09:30:04.106 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:04.104+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:30:04.109 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:04.107+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:30:04.237 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:04.232+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:30:04.238 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:04.234+0000 7f69e572c700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:30:04.296 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:04.293+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:04.297 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:04.294+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:04.549 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:04.547+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:04.553 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:04.548+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:04.619 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:04.617+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:04.620 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:04.618+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:30:04.621 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:04.619+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 8 2021-07-10T09:30:04.765 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:04.763+0000 7f422a97b700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:30:04.766 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:04.764+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:30:05.201 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:30:05.199+0000 7f72c6890700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2021-07-10T09:30:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap8 2021-07-10T09:30:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap8 2021-07-10T09:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap8 2021-07-10T09:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:05.758 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:05.757+0000 7f25794d0700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:30:06.045 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:06.043+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:30:06.045 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:06.043+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:30:06.046 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:06.044+0000 7f69e572c700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2021-07-10T09:30:06.047 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:06.044+0000 7f69e7730700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 7 2021-07-10T09:30:06.048 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:06.044+0000 7f69e672e700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2021-07-10T09:30:06.282 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:06.279+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:06.337 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:06.335+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:06.523 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.522+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:06.581 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.580+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:06.582 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.580+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:30:06.582 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.581+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:30:06.583 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.581+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:30:06.618 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2021-07-10T09:30:06.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2021-07-10T09:30:06.619 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2021-07-10T09:30:06.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2021-07-10T09:30:06.622 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2021-07-10T09:30:06.649 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.648+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:06.650 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.648+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:06.650 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:06.649+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 8 2021-07-10T09:30:06.725 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:06.723+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:06.726 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:06.724+0000 7f4226172700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:06.876 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:06.874+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:07.018 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:07.017+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:30:07.018 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:07.017+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:30:07.083 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:07.081+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:07.084 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:07.081+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:07.153 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:07.152+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:07.526 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2021-07-10T09:30:07.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2021-07-10T09:30:07.527 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2021-07-10T09:30:07.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi105.stderr:2021-07-10T09:30:07.526+0000 7fc71c11a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 40110) UID: 1000 2021-07-10T09:30:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap8-clean 2021-07-10T09:30:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap8-clean 2021-07-10T09:30:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap8-clean 2021-07-10T09:30:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:07.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:30:07.732 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:07.730+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:07.732 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:07.730+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:07.847 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:07.845+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:08.232 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:08.230+0000 7f69e7f31700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:30:08.303 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:08.301+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:08.304 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:08.302+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:30:08.305 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:08.304+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:08.306 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:08.304+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:30:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:08.861 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:30:08.859+0000 7f72ca097700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:30:09.043 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:09.041+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:09.043 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:09.042+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:30:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:09.528 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2021-07-10T09:30:09.528 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:30:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:30:09.529 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2021-07-10T09:30:09.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2021-07-10T09:30:09.530 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2021-07-10T09:30:09.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2021-07-10T09:30:09.533 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2021-07-10T09:30:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap9 2021-07-10T09:30:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap9 2021-07-10T09:30:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap9 2021-07-10T09:30:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:09.651 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:09.650+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:09.652 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:09.650+0000 7f25794d0700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:09.653 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:09.651+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:09.653 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:09.651+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:30:09.946 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:09.945+0000 7f69e7730700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2021-07-10T09:30:09.947 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:09.945+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2021-07-10T09:30:10.117 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:10.115+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:10.117 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:10.115+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:10.169 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:10.168+0000 7f69e3f29700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:30:10.176 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:10.168+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:30:10.274 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:10.273+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:10.492 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:10.490+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:10.915 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:10.913+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:10.995 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:10.993+0000 7f4225170700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2021-07-10T09:30:11.063 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:11.062+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:11.064 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:11.063+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:11.127 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:11.125+0000 7f317a86d700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:11.127 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:11.126+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap9-clean 2021-07-10T09:30:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap9-clean 2021-07-10T09:30:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap9-clean 2021-07-10T09:30:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:11.667 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:11.665+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:30:11.668 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:11.666+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2021-07-10T09:30:11.671 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:11.669+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:30:12.013 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:30:12.011+0000 7f5112a91700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:30:12.068 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:12.066+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:12.068 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:12.066+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:12.226 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:12.224+0000 7f69e472a700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:30:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:12.840 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:12.838+0000 7f422a17a700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:30:12.845 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:12.839+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:30:12.846 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:12.839+0000 7f422496f700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:30:12.920 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:12.919+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:30:13.282 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:13.280+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:13.282 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:13.280+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:13.354 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:13.353+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap10 2021-07-10T09:30:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap10 2021-07-10T09:30:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap10 2021-07-10T09:30:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:13.869 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:13.866+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:13.869 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:13.867+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:13.869 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:13.867+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:13.870 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:13.868+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:30:13.870 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:13.868+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:30:14.032 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:14.030+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:14.032 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:14.030+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:14.193 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:14.191+0000 7f69ea736700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:30:14.382 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:14.381+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2021-07-10T09:30:14.455 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:14.453+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:14.456 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:14.454+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:14.456 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:14.454+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:14.456 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:14.454+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:14.612 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:14.610+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:14.612 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:14.610+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:14.731 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:14.729+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:14.732 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:14.730+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:14.732 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:14.730+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:14.797 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:14.796+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:14.876 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:14.874+0000 7f422a17a700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2021-07-10T09:30:14.877 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:14.875+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 8 2021-07-10T09:30:15.019 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:15.017+0000 7f422396d700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:15.020 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:15.017+0000 7f4227174700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:30:15.020 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:15.017+0000 7f4225170700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2021-07-10T09:30:15.093 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:15.091+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:30:15.093 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:15.091+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:15.100 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:15.094+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:15.169 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:15.168+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:15.258 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:15.255+0000 7f317385f700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap10-clean 2021-07-10T09:30:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror test snap10-clean 2021-07-10T09:30:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap10-clean 2021-07-10T09:30:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:30:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export 2021-07-10T09:30:15.861 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:15.859+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:15.861 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:15.859+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:15.861 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:15.859+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:15.862 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:15.859+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:30:15.863 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:15.862+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:30:15.978 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:15.976+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:30:16.143 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:30:16.141+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:30:16.144 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:30:16.141+0000 7f5116a99700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:30:16.199 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:16.197+0000 7f69e9734700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:30:16.199 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:16.197+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:30:16.200 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:16.197+0000 7f69e472a700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:30:16.266 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:16.264+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:30:16.267 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:16.265+0000 7f69e572c700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2021-07-10T09:30:16.321 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:16.319+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:16.322 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:16.320+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:30:16.322 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:30:16.321+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:30:16.388 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:16.387+0000 7f69e8f33700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 7 2021-07-10T09:30:16.392 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:30:16.388+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 8 2021-07-10T09:30:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:16.633 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:30:16.631+0000 7f72c7091700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2021-07-10T09:30:16.634 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:30:16.631+0000 7f72ca898700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2021-07-10T09:30:16.634 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:30:16.631+0000 7f72c608f700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2021-07-10T09:30:16.634 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:30:16.631+0000 7f72c7892700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:30:16.735 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:16.732+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:16.736 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:16.733+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:16.736 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:16.733+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:16.736 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:16.734+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:16.737 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:16.735+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:30:16.801 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:16.799+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:16.938 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:30:16.936+0000 7f422a97b700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:30:17.078 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:17.077+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:30:17.079 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:17.078+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:30:17.080 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:17.079+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:30:17.354 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:30:17.352+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:30:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2021-07-10T09:30:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-test.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-test.export 2021-07-10T09:30:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 10 2021-07-10T09:30:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap1 2021-07-10T09:30:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap1 2021-07-10T09:30:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap1 2021-07-10T09:30:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2021-07-10T09:30:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap1-clean 2021-07-10T09:30:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2021-07-10T09:30:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap1-clean 2021-07-10T09:30:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2021-07-10T09:30:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap2 2021-07-10T09:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap2 2021-07-10T09:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap2 2021-07-10T09:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2021-07-10T09:30:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap2-clean 2021-07-10T09:30:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2021-07-10T09:30:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap2-clean 2021-07-10T09:30:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2021-07-10T09:30:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap3 2021-07-10T09:30:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap3 2021-07-10T09:30:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap3 2021-07-10T09:30:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2021-07-10T09:30:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap3-clean 2021-07-10T09:30:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2021-07-10T09:30:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap3-clean 2021-07-10T09:30:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2021-07-10T09:30:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap4 2021-07-10T09:30:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap4 2021-07-10T09:30:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap4 2021-07-10T09:30:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2021-07-10T09:30:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap4-clean 2021-07-10T09:30:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2021-07-10T09:30:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap4-clean 2021-07-10T09:30:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2021-07-10T09:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap5 2021-07-10T09:30:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap5 2021-07-10T09:30:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap5 2021-07-10T09:30:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2021-07-10T09:30:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap5-clean 2021-07-10T09:30:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2021-07-10T09:30:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap5-clean 2021-07-10T09:30:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2021-07-10T09:30:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap6 2021-07-10T09:30:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap6 2021-07-10T09:30:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap6 2021-07-10T09:30:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2021-07-10T09:30:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap6-clean 2021-07-10T09:30:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2021-07-10T09:30:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap6-clean 2021-07-10T09:30:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2021-07-10T09:30:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap7 2021-07-10T09:30:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap7 2021-07-10T09:30:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap7 2021-07-10T09:30:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2021-07-10T09:30:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap7-clean 2021-07-10T09:30:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2021-07-10T09:30:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap7-clean 2021-07-10T09:30:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2021-07-10T09:30:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap8 2021-07-10T09:30:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap8 2021-07-10T09:30:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap8 2021-07-10T09:30:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2021-07-10T09:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap8-clean 2021-07-10T09:30:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2021-07-10T09:30:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap8-clean 2021-07-10T09:30:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2021-07-10T09:30:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap9 2021-07-10T09:30:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap9 2021-07-10T09:30:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap9 2021-07-10T09:30:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2021-07-10T09:30:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap9-clean 2021-07-10T09:30:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2021-07-10T09:30:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap9-clean 2021-07-10T09:30:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2021-07-10T09:30:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:30:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap10 2021-07-10T09:30:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap10 2021-07-10T09:30:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap10 2021-07-10T09:30:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2021-07-10T09:30:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap10-clean 2021-07-10T09:30:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2021-07-10T09:30:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap10-clean 2021-07-10T09:30:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2021-07-10T09:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:30:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror test 2021-07-10T09:30:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:30:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:30:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror test 2021-07-10T09:30:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:30:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2021-07-10T09:30:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/test 2021-07-10T09:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror test deleted 2021-07-10T09:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:30:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:30:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:30:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:30:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:30:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror test 2021-07-10T09:30:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:30:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:30:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=test 2021-07-10T09:30:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/test 2021-07-10T09:30:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:30:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=10a95c32b3f8 2021-07-10T09:30:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:30:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:30:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror test deleted 10a95c32b3f8 2021-07-10T09:30:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=10a95c32b3f8 2021-07-10T09:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror test 2021-07-10T09:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=test 2021-07-10T09:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/test 2021-07-10T09:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:30:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=10a95c32b3f8 2021-07-10T09:30:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 10a95c32b3f8 2021-07-10T09:30:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 10a95c32b3f8 -o 10a95c32b3f8 = 10a95c32b3f8 2021-07-10T09:30:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:30:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = present 2021-07-10T09:30:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:30:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:30:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror test deleted 10a95c32b3f8 2021-07-10T09:30:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:30:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:30:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=10a95c32b3f8 2021-07-10T09:30:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:30:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:30:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror test 2021-07-10T09:30:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:30:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:30:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=test 2021-07-10T09:30:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/test 2021-07-10T09:30:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:30:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=10a95c32b3f8 2021-07-10T09:30:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 10a95c32b3f8 2021-07-10T09:30:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 10a95c32b3f8 -o 10a95c32b3f8 = 10a95c32b3f8 2021-07-10T09:30:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:30:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = present 2021-07-10T09:30:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:30:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:30:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror test deleted 10a95c32b3f8 2021-07-10T09:30:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:30:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:30:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=10a95c32b3f8 2021-07-10T09:30:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:30:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:30:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror test 2021-07-10T09:30:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:30:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:30:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=test 2021-07-10T09:30:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/test 2021-07-10T09:30:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:30:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=10a95c32b3f8 2021-07-10T09:30:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 10a95c32b3f8 2021-07-10T09:30:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 10a95c32b3f8 -o 10a95c32b3f8 = 10a95c32b3f8 2021-07-10T09:30:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:30:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = present 2021-07-10T09:30:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:30:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:30:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror test deleted 10a95c32b3f8 2021-07-10T09:30:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:30:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:30:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=10a95c32b3f8 2021-07-10T09:30:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:30:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:30:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror test 2021-07-10T09:30:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:30:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:30:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=test 2021-07-10T09:30:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/test 2021-07-10T09:30:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:30:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=10a95c32b3f8 2021-07-10T09:30:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 10a95c32b3f8 2021-07-10T09:30:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 10a95c32b3f8 -o 10a95c32b3f8 = 10a95c32b3f8 2021-07-10T09:30:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:30:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = present 2021-07-10T09:30:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:30:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:30:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror test deleted 10a95c32b3f8 2021-07-10T09:30:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:30:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:30:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:30:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:30:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=10a95c32b3f8 2021-07-10T09:30:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror test 2021-07-10T09:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:30:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=test 2021-07-10T09:30:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/test 2021-07-10T09:30:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:30:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=10a95c32b3f8 2021-07-10T09:30:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 10a95c32b3f8 2021-07-10T09:30:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 10a95c32b3f8 -o 10a95c32b3f8 = 10a95c32b3f8 2021-07-10T09:30:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:30:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = present 2021-07-10T09:30:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:31:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror test deleted 10a95c32b3f8 2021-07-10T09:31:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:31:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=test 2021-07-10T09:31:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:31:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=10a95c32b3f8 2021-07-10T09:31:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:31:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:31:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror test 2021-07-10T09:31:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:31:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:31:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=test 2021-07-10T09:31:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/test 2021-07-10T09:31:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:31:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image test: (2) No such file or directory 2021-07-10T09:31:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:31:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:31:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:31:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:31:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ testlog 'TEST: create many images' 2021-07-10T09:31:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee -a /tmp/tmp.VSH8FeaLBn/rbd-mirror.test.log 2021-07-10T09:31:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ date '+%F %T' 2021-07-10T09:31:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo 2021-07-10 09:31:02 TEST: create many images 2021-07-10T09:31:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10 09:31:02 TEST: create many images 2021-07-10T09:31:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ snap_name=snap 2021-07-10T09:31:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 50 2021-07-10T09:31:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:31:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_1 2021-07-10T09:31:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2021-07-10T09:31:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:31:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:31:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_1 128M 2021-07-10T09:31:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:31:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:31:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:31:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:31:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2021-07-10T09:31:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:31:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2021-07-10T09:31:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:31:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2021-07-10T09:31:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:31:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:31:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_1 2021-07-10T09:31:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:31:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:31:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:31:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:31:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:31:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 51428352 bytes 2021-07-10T09:31:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 52042752 bytes 2021-07-10T09:31:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 96314368 bytes 2021-07-10T09:31:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 125170688 bytes 2021-07-10T09:31:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 229619712 bytes 2021-07-10T09:31:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 262873088 bytes 2021-07-10T09:31:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 307357696 bytes 2021-07-10T09:31:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 405572608 bytes 2021-07-10T09:31:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 436614144 bytes 2021-07-10T09:31:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 453038080 bytes 2021-07-10T09:31:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 522741760 bytes 2021-07-10T09:31:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 550148096 bytes 2021-07-10T09:31:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 561979392 bytes 2021-07-10T09:31:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 605840384 bytes 2021-07-10T09:31:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 612587520 bytes 2021-07-10T09:31:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 681282560 bytes 2021-07-10T09:31:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 712116224 bytes 2021-07-10T09:31:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 743546880 bytes 2021-07-10T09:31:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 759311360 bytes 2021-07-10T09:31:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 808055808 bytes 2021-07-10T09:31:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 883528704 bytes 2021-07-10T09:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 908369920 bytes 2021-07-10T09:31:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 941127680 bytes 2021-07-10T09:31:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 969985024 bytes 2021-07-10T09:31:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1042080768 bytes 2021-07-10T09:31:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1109686272 bytes 2021-07-10T09:31:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1124721664 bytes 2021-07-10T09:31:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1171026944 bytes 2021-07-10T09:31:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1198087168 bytes 2021-07-10T09:31:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1208589312 bytes 2021-07-10T09:31:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1277599744 bytes 2021-07-10T09:31:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1327648768 bytes 2021-07-10T09:31:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1346846720 bytes 2021-07-10T09:31:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1357724672 bytes 2021-07-10T09:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1390257152 bytes 2021-07-10T09:31:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1430547456 bytes 2021-07-10T09:31:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1458181120 bytes 2021-07-10T09:31:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1485659136 bytes 2021-07-10T09:31:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1499625472 bytes 2021-07-10T09:31:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1533333504 bytes 2021-07-10T09:31:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1553485824 bytes 2021-07-10T09:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1567751168 bytes 2021-07-10T09:31:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1596546048 bytes 2021-07-10T09:31:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1622759424 bytes 2021-07-10T09:31:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1735695360 bytes 2021-07-10T09:31:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 1789784064 bytes 2021-07-10T09:31:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 1819561984 bytes 2021-07-10T09:31:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:31:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:31:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:31:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:31:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:31:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_2 2021-07-10T09:31:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2021-07-10T09:31:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:31:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:31:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_2 128M 2021-07-10T09:31:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:31:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:31:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:31:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:31:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2021-07-10T09:31:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:31:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2021-07-10T09:31:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:31:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2021-07-10T09:31:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:31:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:31:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_2 2021-07-10T09:31:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:31:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:31:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:31:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:31:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:31:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 621568 bytes 2021-07-10T09:31:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 21840896 bytes 2021-07-10T09:31:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 64414720 bytes 2021-07-10T09:31:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 78707712 bytes 2021-07-10T09:31:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 90011648 bytes 2021-07-10T09:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 113797120 bytes 2021-07-10T09:31:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 169458688 bytes 2021-07-10T09:31:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 195616768 bytes 2021-07-10T09:31:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 224750592 bytes 2021-07-10T09:31:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 263021568 bytes 2021-07-10T09:31:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 344554496 bytes 2021-07-10T09:31:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 366765056 bytes 2021-07-10T09:31:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 388041728 bytes 2021-07-10T09:31:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 406562816 bytes 2021-07-10T09:31:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 446361600 bytes 2021-07-10T09:31:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 486679552 bytes 2021-07-10T09:31:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 526823424 bytes 2021-07-10T09:31:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 559772672 bytes 2021-07-10T09:31:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 603029504 bytes 2021-07-10T09:31:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 641595392 bytes 2021-07-10T09:31:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 662756352 bytes 2021-07-10T09:31:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:31:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:31:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:31:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:31:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:31:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_3 2021-07-10T09:31:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2021-07-10T09:31:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:31:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:31:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_3 128M 2021-07-10T09:31:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:31:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:31:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:31:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:31:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2021-07-10T09:31:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:31:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2021-07-10T09:31:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:31:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2021-07-10T09:31:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:31:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:31:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_3 2021-07-10T09:31:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:31:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:31:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:31:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:31:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:31:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 41585664 bytes 2021-07-10T09:31:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 72850432 bytes 2021-07-10T09:31:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 111552512 bytes 2021-07-10T09:31:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 207346688 bytes 2021-07-10T09:31:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 239240192 bytes 2021-07-10T09:31:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 244791296 bytes 2021-07-10T09:31:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 319675392 bytes 2021-07-10T09:31:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 324608000 bytes 2021-07-10T09:31:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 372784128 bytes 2021-07-10T09:31:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 421043200 bytes 2021-07-10T09:31:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 532687872 bytes 2021-07-10T09:31:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 610267136 bytes 2021-07-10T09:31:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 621404160 bytes 2021-07-10T09:31:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 666563584 bytes 2021-07-10T09:31:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 715985920 bytes 2021-07-10T09:31:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 716821504 bytes 2021-07-10T09:31:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 739116032 bytes 2021-07-10T09:31:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 780678144 bytes 2021-07-10T09:31:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 859337728 bytes 2021-07-10T09:31:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 902428672 bytes 2021-07-10T09:31:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 918185984 bytes 2021-07-10T09:31:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 935205888 bytes 2021-07-10T09:31:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 962910208 bytes 2021-07-10T09:31:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 969180160 bytes 2021-07-10T09:31:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1015105536 bytes 2021-07-10T09:31:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1079506944 bytes 2021-07-10T09:31:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1163246592 bytes 2021-07-10T09:31:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1223156736 bytes 2021-07-10T09:31:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1241176064 bytes 2021-07-10T09:31:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1249841152 bytes 2021-07-10T09:31:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1287279616 bytes 2021-07-10T09:31:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1293511680 bytes 2021-07-10T09:31:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1324125184 bytes 2021-07-10T09:31:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1363593216 bytes 2021-07-10T09:31:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1371732992 bytes 2021-07-10T09:31:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1418725376 bytes 2021-07-10T09:31:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1490734080 bytes 2021-07-10T09:31:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:31:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:31:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:31:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:31:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:31:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_4 2021-07-10T09:31:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2021-07-10T09:31:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:31:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:31:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_4 128M 2021-07-10T09:31:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:31:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:31:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:31:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:31:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2021-07-10T09:31:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:31:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2021-07-10T09:31:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:31:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2021-07-10T09:31:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:31:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_4 2021-07-10T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:31:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:31:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=13 2021-07-10T09:31:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:31:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:31:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 62120960 bytes 2021-07-10T09:31:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 100618240 bytes 2021-07-10T09:31:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 103166976 bytes 2021-07-10T09:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 121255936 bytes 2021-07-10T09:31:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 228310016 bytes 2021-07-10T09:31:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 298808320 bytes 2021-07-10T09:31:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 306860032 bytes 2021-07-10T09:31:35.623 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2021-07-10T09:31:35.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2021-07-10T09:31:35.624 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2021-07-10T09:31:35.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:31:35.624+0000 7f5c13e8b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 45009) UID: 1000 2021-07-10T09:31:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 347870208 bytes 2021-07-10T09:31:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 359967744 bytes 2021-07-10T09:31:36.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:31:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 439809024 bytes 2021-07-10T09:31:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 472219648 bytes 2021-07-10T09:31:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 515966976 bytes 2021-07-10T09:31:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 540110848 bytes 2021-07-10T09:31:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 560464896 bytes 2021-07-10T09:31:37.407 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:31:37.405+0000 7f25794d0700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:31:37.408 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:31:37.406+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:31:37.416 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:31:37.416+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 12 2021-07-10T09:31:37.417 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:31:37.416+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:31:37.419 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:31:37.419+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:31:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 706109440 bytes 2021-07-10T09:31:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 733536256 bytes 2021-07-10T09:31:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 741729280 bytes 2021-07-10T09:31:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 774738944 bytes 2021-07-10T09:31:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 795358208 bytes 2021-07-10T09:31:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 800377856 bytes 2021-07-10T09:31:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 814474240 bytes 2021-07-10T09:31:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 839868416 bytes 2021-07-10T09:31:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 935844864 bytes 2021-07-10T09:31:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1023637504 bytes 2021-07-10T09:31:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1030317056 bytes 2021-07-10T09:31:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1138974720 bytes 2021-07-10T09:31:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1210046464 bytes 2021-07-10T09:31:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1242406912 bytes 2021-07-10T09:31:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1298698240 bytes 2021-07-10T09:31:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1341755392 bytes 2021-07-10T09:31:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1342334976 bytes 2021-07-10T09:31:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1394131968 bytes 2021-07-10T09:31:42.694 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:31:42.692+0000 7f69e4f2b700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:31:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1469794304 bytes 2021-07-10T09:31:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1533485056 bytes 2021-07-10T09:31:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1587414016 bytes 2021-07-10T09:31:43.625 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2021-07-10T09:31:43.626 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:31:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:31:43.627 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2021-07-10T09:31:43.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:31:43.628 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:31:43.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:31:43.631 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2021-07-10T09:31:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1629903872 bytes 2021-07-10T09:31:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1651873792 bytes 2021-07-10T09:31:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1679869952 bytes 2021-07-10T09:31:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1745239040 bytes 2021-07-10T09:31:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1768044544 bytes 2021-07-10T09:31:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1768629248 bytes 2021-07-10T09:31:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1795486720 bytes 2021-07-10T09:31:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1842746368 bytes 2021-07-10T09:31:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1897953280 bytes 2021-07-10T09:31:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1942147072 bytes 2021-07-10T09:31:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 1957009408 bytes 2021-07-10T09:31:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 1991601152 bytes 2021-07-10T09:31:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:31:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:31:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:31:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:31:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:31:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_5 2021-07-10T09:31:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2021-07-10T09:31:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:31:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:31:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_5 128M 2021-07-10T09:31:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:31:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:31:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:31:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:31:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2021-07-10T09:31:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:31:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2021-07-10T09:31:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:31:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2021-07-10T09:31:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:31:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:31:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_5 2021-07-10T09:31:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:31:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:31:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=7 2021-07-10T09:31:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:31:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:31:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 116079616 bytes 2021-07-10T09:31:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 125396992 bytes 2021-07-10T09:31:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 153764864 bytes 2021-07-10T09:31:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 187115520 bytes 2021-07-10T09:31:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 231248896 bytes 2021-07-10T09:31:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 273984512 bytes 2021-07-10T09:31:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 306068480 bytes 2021-07-10T09:31:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 374345728 bytes 2021-07-10T09:31:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 412478464 bytes 2021-07-10T09:31:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 466465792 bytes 2021-07-10T09:31:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 467081216 bytes 2021-07-10T09:31:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 510110720 bytes 2021-07-10T09:31:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 537061376 bytes 2021-07-10T09:31:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 570150912 bytes 2021-07-10T09:31:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 570778624 bytes 2021-07-10T09:31:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 614647808 bytes 2021-07-10T09:31:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 685903872 bytes 2021-07-10T09:31:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 697847808 bytes 2021-07-10T09:31:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 720571392 bytes 2021-07-10T09:31:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 765488128 bytes 2021-07-10T09:31:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 820319232 bytes 2021-07-10T09:31:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 876575744 bytes 2021-07-10T09:31:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 903372800 bytes 2021-07-10T09:31:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 964516864 bytes 2021-07-10T09:31:54.534 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4490478643849989) > weight (0.25) 2021-07-10T09:31:54.534 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4987946800443199) > weight (0.25) 2021-07-10T09:31:54.534 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7376600648201874) > weight (0.25) 2021-07-10T09:31:54.535 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2021-07-10T09:31:54.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2021-07-10T09:31:54.535 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2021-07-10T09:31:54.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi105.stderr:2021-07-10T09:31:54.535+0000 7f0f915b4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36725) UID: 1000 2021-07-10T09:31:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 999478272 bytes 2021-07-10T09:31:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:31:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:31:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:31:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:31:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:31:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_6 2021-07-10T09:31:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2021-07-10T09:31:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:31:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:31:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_6 128M 2021-07-10T09:31:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:31:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:31:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:31:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:31:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:31:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2021-07-10T09:31:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:31:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2021-07-10T09:31:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:31:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:31:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2021-07-10T09:31:54.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:31:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:31:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:31:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_6 2021-07-10T09:31:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:31:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:31:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:31:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:31:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:31:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:31:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:31:55.684 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:31:55.682+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:31:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 101854208 bytes 2021-07-10T09:31:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 207108096 bytes 2021-07-10T09:31:56.538 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2021-07-10T09:31:56.539 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:31:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:31:56.540 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2021-07-10T09:31:56.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2021-07-10T09:31:56.540 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2021-07-10T09:31:56.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2021-07-10T09:31:56.547 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2021-07-10T09:31:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 285187072 bytes 2021-07-10T09:31:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 338635776 bytes 2021-07-10T09:31:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 352942080 bytes 2021-07-10T09:31:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 388690944 bytes 2021-07-10T09:31:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 423115776 bytes 2021-07-10T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 476025856 bytes 2021-07-10T09:31:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 535687168 bytes 2021-07-10T09:31:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 549994496 bytes 2021-07-10T09:31:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 565975040 bytes 2021-07-10T09:31:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 597517312 bytes 2021-07-10T09:31:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 637573120 bytes 2021-07-10T09:31:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 679353344 bytes 2021-07-10T09:32:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 698022912 bytes 2021-07-10T09:32:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 782363648 bytes 2021-07-10T09:32:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 811719680 bytes 2021-07-10T09:32:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 840530944 bytes 2021-07-10T09:32:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 911360000 bytes 2021-07-10T09:32:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 1016836096 bytes 2021-07-10T09:32:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 1039089664 bytes 2021-07-10T09:32:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 1067579392 bytes 2021-07-10T09:32:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1111745536 bytes 2021-07-10T09:32:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1124847616 bytes 2021-07-10T09:32:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1195276288 bytes 2021-07-10T09:32:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:32:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:32:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:32:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:32:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:32:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_7 2021-07-10T09:32:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2021-07-10T09:32:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:32:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:32:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_7 128M 2021-07-10T09:32:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:32:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:32:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:32:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:32:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2021-07-10T09:32:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:32:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2021-07-10T09:32:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:32:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2021-07-10T09:32:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:32:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:32:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_7 2021-07-10T09:32:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:32:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:32:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=6 2021-07-10T09:32:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:32:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:32:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 16048128 bytes 2021-07-10T09:32:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 16785408 bytes 2021-07-10T09:32:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 70771712 bytes 2021-07-10T09:32:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 89639936 bytes 2021-07-10T09:32:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 116975616 bytes 2021-07-10T09:32:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 149440512 bytes 2021-07-10T09:32:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 182661120 bytes 2021-07-10T09:32:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 216898560 bytes 2021-07-10T09:32:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 258728960 bytes 2021-07-10T09:32:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 326672384 bytes 2021-07-10T09:32:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 369691648 bytes 2021-07-10T09:32:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 370309120 bytes 2021-07-10T09:32:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 412659712 bytes 2021-07-10T09:32:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 413289472 bytes 2021-07-10T09:32:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 429618176 bytes 2021-07-10T09:32:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 440590336 bytes 2021-07-10T09:32:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 464570368 bytes 2021-07-10T09:32:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 538817536 bytes 2021-07-10T09:32:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 576239616 bytes 2021-07-10T09:32:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 629659648 bytes 2021-07-10T09:32:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 664766464 bytes 2021-07-10T09:32:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 714456064 bytes 2021-07-10T09:32:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:32:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:32:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:32:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:32:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:32:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_8 2021-07-10T09:32:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2021-07-10T09:32:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:32:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:32:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_8 128M 2021-07-10T09:32:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:32:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:32:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:32:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:32:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2021-07-10T09:32:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2021-07-10T09:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:32:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2021-07-10T09:32:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_8 2021-07-10T09:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:32:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:32:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:32:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:32:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:32:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 634880 bytes 2021-07-10T09:32:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 57091072 bytes 2021-07-10T09:32:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 79279104 bytes 2021-07-10T09:32:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 91957248 bytes 2021-07-10T09:32:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 127620096 bytes 2021-07-10T09:32:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 254356480 bytes 2021-07-10T09:32:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 282443776 bytes 2021-07-10T09:32:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 357614592 bytes 2021-07-10T09:32:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 434292736 bytes 2021-07-10T09:32:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 462102528 bytes 2021-07-10T09:32:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 462715904 bytes 2021-07-10T09:32:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 551116800 bytes 2021-07-10T09:32:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 604116992 bytes 2021-07-10T09:32:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 647182336 bytes 2021-07-10T09:32:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 650800128 bytes 2021-07-10T09:32:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 674933760 bytes 2021-07-10T09:32:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 689171456 bytes 2021-07-10T09:32:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:32:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:32:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:32:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:32:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:32:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_9 2021-07-10T09:32:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2021-07-10T09:32:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:32:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:32:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_9 128M 2021-07-10T09:32:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:32:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:32:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:32:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:32:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2021-07-10T09:32:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:32:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2021-07-10T09:32:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:32:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2021-07-10T09:32:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:32:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:32:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_9 2021-07-10T09:32:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:32:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:32:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:32:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:32:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:32:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 26526720 bytes 2021-07-10T09:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 27134976 bytes 2021-07-10T09:32:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 89674752 bytes 2021-07-10T09:32:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 154862592 bytes 2021-07-10T09:32:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 197398528 bytes 2021-07-10T09:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 250245120 bytes 2021-07-10T09:32:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 285448192 bytes 2021-07-10T09:32:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 313775104 bytes 2021-07-10T09:32:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 423231488 bytes 2021-07-10T09:32:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 494535680 bytes 2021-07-10T09:32:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 542212096 bytes 2021-07-10T09:32:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 589758464 bytes 2021-07-10T09:32:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 602997760 bytes 2021-07-10T09:32:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 606860288 bytes 2021-07-10T09:32:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 670563328 bytes 2021-07-10T09:32:22.475 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:32:22.474+0000 7f5116298700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:32:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 747098112 bytes 2021-07-10T09:32:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 787216384 bytes 2021-07-10T09:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 787831808 bytes 2021-07-10T09:32:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 828678144 bytes 2021-07-10T09:32:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 846110720 bytes 2021-07-10T09:32:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 910857216 bytes 2021-07-10T09:32:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 956575744 bytes 2021-07-10T09:32:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 975287296 bytes 2021-07-10T09:32:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1080343552 bytes 2021-07-10T09:32:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1099643904 bytes 2021-07-10T09:32:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1145687040 bytes 2021-07-10T09:32:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:32:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:32:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:32:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:32:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:32:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_10 2021-07-10T09:32:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2021-07-10T09:32:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:32:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:32:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_10 128M 2021-07-10T09:32:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:32:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2021-07-10T09:32:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:32:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2021-07-10T09:32:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:32:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2021-07-10T09:32:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:32:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:32:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_10 2021-07-10T09:32:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:32:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:32:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:32:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:32:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:32:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 112745472 bytes 2021-07-10T09:32:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 122519552 bytes 2021-07-10T09:32:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 162100224 bytes 2021-07-10T09:32:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 211430400 bytes 2021-07-10T09:32:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 218625024 bytes 2021-07-10T09:32:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 244194304 bytes 2021-07-10T09:32:28.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 314615808 bytes 2021-07-10T09:32:28.900 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:32:28.899+0000 7f69e9f35700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:32:28.901 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:32:28.899+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:32:28.901 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:32:28.899+0000 7f69e5f2d700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:32:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 365368320 bytes 2021-07-10T09:32:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 421772288 bytes 2021-07-10T09:32:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 477240320 bytes 2021-07-10T09:32:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 534841344 bytes 2021-07-10T09:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 552940544 bytes 2021-07-10T09:32:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 568908800 bytes 2021-07-10T09:32:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 628791296 bytes 2021-07-10T09:32:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 694251520 bytes 2021-07-10T09:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:32:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:32:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_11 2021-07-10T09:32:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2021-07-10T09:32:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:32:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:32:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_11 128M 2021-07-10T09:32:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:32:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:32:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:32:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:32:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2021-07-10T09:32:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:32:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2021-07-10T09:32:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:32:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2021-07-10T09:32:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:32:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:32:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_11 2021-07-10T09:32:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:32:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:32:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:32:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:32:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:32:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 623616 bytes 2021-07-10T09:32:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 49659904 bytes 2021-07-10T09:32:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 68301824 bytes 2021-07-10T09:32:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 137245696 bytes 2021-07-10T09:32:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 210989056 bytes 2021-07-10T09:32:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 257111040 bytes 2021-07-10T09:32:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 285559808 bytes 2021-07-10T09:32:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 313925632 bytes 2021-07-10T09:32:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 364088320 bytes 2021-07-10T09:32:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 414440448 bytes 2021-07-10T09:32:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 471561216 bytes 2021-07-10T09:32:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 512241664 bytes 2021-07-10T09:32:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 537345024 bytes 2021-07-10T09:32:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 574434304 bytes 2021-07-10T09:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 655771648 bytes 2021-07-10T09:32:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:32:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:32:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:32:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:32:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:32:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_12 2021-07-10T09:32:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2021-07-10T09:32:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:32:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:32:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_12 128M 2021-07-10T09:32:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:32:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:32:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:32:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:32:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2021-07-10T09:32:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:32:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2021-07-10T09:32:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:32:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2021-07-10T09:32:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:32:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:32:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_12 2021-07-10T09:32:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:32:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=13 2021-07-10T09:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:32:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:32:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 27110400 bytes 2021-07-10T09:32:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 71674880 bytes 2021-07-10T09:32:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 88790016 bytes 2021-07-10T09:32:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 111212544 bytes 2021-07-10T09:32:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 126540800 bytes 2021-07-10T09:32:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 155332608 bytes 2021-07-10T09:32:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 221700096 bytes 2021-07-10T09:32:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 260341760 bytes 2021-07-10T09:32:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 296314880 bytes 2021-07-10T09:32:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 327662592 bytes 2021-07-10T09:32:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 446028800 bytes 2021-07-10T09:32:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 466937856 bytes 2021-07-10T09:32:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 518616064 bytes 2021-07-10T09:32:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 557871104 bytes 2021-07-10T09:32:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 603776000 bytes 2021-07-10T09:32:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 646180864 bytes 2021-07-10T09:32:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 702435328 bytes 2021-07-10T09:32:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 756515840 bytes 2021-07-10T09:32:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 771531776 bytes 2021-07-10T09:32:44.633 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2021-07-10T09:32:44.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2021-07-10T09:32:44.634 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2021-07-10T09:32:44.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:32:44.636+0000 7f7d8d9e2700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 47758) UID: 1000 2021-07-10T09:32:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 839068672 bytes 2021-07-10T09:32:45.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:32:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 863561728 bytes 2021-07-10T09:32:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 892963840 bytes 2021-07-10T09:32:45.635 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2021-07-10T09:32:45.635 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:32:45.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:32:45.636 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2021-07-10T09:32:45.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:32:45.637 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:32:45.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:32:45.641 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2021-07-10T09:32:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 967719936 bytes 2021-07-10T09:32:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 996432896 bytes 2021-07-10T09:32:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1020018688 bytes 2021-07-10T09:32:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1058438144 bytes 2021-07-10T09:32:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1105241088 bytes 2021-07-10T09:32:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1147036672 bytes 2021-07-10T09:32:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1172157440 bytes 2021-07-10T09:32:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1207870464 bytes 2021-07-10T09:32:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1241199616 bytes 2021-07-10T09:32:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1256394752 bytes 2021-07-10T09:32:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1325338624 bytes 2021-07-10T09:32:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1345595392 bytes 2021-07-10T09:32:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1346177024 bytes 2021-07-10T09:32:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1407601664 bytes 2021-07-10T09:32:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1425417216 bytes 2021-07-10T09:32:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1482293248 bytes 2021-07-10T09:32:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1517355008 bytes 2021-07-10T09:32:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1552670720 bytes 2021-07-10T09:32:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1651003392 bytes 2021-07-10T09:32:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1651619840 bytes 2021-07-10T09:32:51.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:32:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:32:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:32:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:32:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:32:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_13 2021-07-10T09:32:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2021-07-10T09:32:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:32:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:32:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_13 128M 2021-07-10T09:32:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:32:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:32:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:32:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:32:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:32:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2021-07-10T09:32:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:32:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2021-07-10T09:32:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:32:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:32:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2021-07-10T09:32:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:32:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:32:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_13 2021-07-10T09:32:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:32:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:32:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:32:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:32:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:32:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:32:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:32:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 66909184 bytes 2021-07-10T09:32:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 122984448 bytes 2021-07-10T09:32:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 136008704 bytes 2021-07-10T09:32:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 184147968 bytes 2021-07-10T09:32:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 202076160 bytes 2021-07-10T09:32:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 220056576 bytes 2021-07-10T09:32:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 303680512 bytes 2021-07-10T09:32:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 367865856 bytes 2021-07-10T09:32:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 397914112 bytes 2021-07-10T09:32:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 439672832 bytes 2021-07-10T09:32:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 491322368 bytes 2021-07-10T09:32:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 516859904 bytes 2021-07-10T09:32:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 578494464 bytes 2021-07-10T09:32:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 585533440 bytes 2021-07-10T09:32:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 613062656 bytes 2021-07-10T09:32:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 614715392 bytes 2021-07-10T09:32:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 645241856 bytes 2021-07-10T09:32:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 679793664 bytes 2021-07-10T09:32:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 705565696 bytes 2021-07-10T09:32:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 752849920 bytes 2021-07-10T09:32:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 791728128 bytes 2021-07-10T09:32:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 858065920 bytes 2021-07-10T09:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 942698496 bytes 2021-07-10T09:33:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 996927488 bytes 2021-07-10T09:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:33:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:33:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:33:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:33:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:33:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_14 2021-07-10T09:33:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2021-07-10T09:33:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:33:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:33:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_14 128M 2021-07-10T09:33:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:33:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:33:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:33:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:33:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2021-07-10T09:33:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:33:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2021-07-10T09:33:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:33:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2021-07-10T09:33:01.548 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.92159949177353) > weight (0.25) 2021-07-10T09:33:01.548 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.45892216352602255) > weight (0.25) 2021-07-10T09:33:01.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6180630838165068) > weight (0.25) 2021-07-10T09:33:01.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7090984131691238) > weight (0.25) 2021-07-10T09:33:01.550 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2021-07-10T09:33:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:33:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:33:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_14 2021-07-10T09:33:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:33:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:33:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=14 2021-07-10T09:33:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:33:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:33:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 21864448 bytes 2021-07-10T09:33:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 30487552 bytes 2021-07-10T09:33:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 39523328 bytes 2021-07-10T09:33:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 93345792 bytes 2021-07-10T09:33:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 155005952 bytes 2021-07-10T09:33:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 214338560 bytes 2021-07-10T09:33:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 314970112 bytes 2021-07-10T09:33:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 334036992 bytes 2021-07-10T09:33:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 394834944 bytes 2021-07-10T09:33:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 492685312 bytes 2021-07-10T09:33:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 518287360 bytes 2021-07-10T09:33:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 539604992 bytes 2021-07-10T09:33:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 592132096 bytes 2021-07-10T09:33:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 616022016 bytes 2021-07-10T09:33:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 670387200 bytes 2021-07-10T09:33:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 680582144 bytes 2021-07-10T09:33:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 737373184 bytes 2021-07-10T09:33:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 793448448 bytes 2021-07-10T09:33:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 842994688 bytes 2021-07-10T09:33:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 923538432 bytes 2021-07-10T09:33:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 1013816320 bytes 2021-07-10T09:33:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 1149026304 bytes 2021-07-10T09:33:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1180451840 bytes 2021-07-10T09:33:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1181968384 bytes 2021-07-10T09:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1291645952 bytes 2021-07-10T09:33:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1355675648 bytes 2021-07-10T09:33:10.979 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:10.974+0000 7f69f674e700 -1 reset not still connected to 0x5630753f2f70 2021-07-10T09:33:10.980 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:10.974+0000 7f69f674e700 -1 reset not still connected to 0x5630753f3380 2021-07-10T09:33:10.980 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:10.974+0000 7f69f674e700 -1 reset not still connected to 0x5630753f3790 2021-07-10T09:33:10.980 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:10.976+0000 7f69f674e700 -1 reset not still connected to 0x563075db2c30 2021-07-10T09:33:10.981 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:10.976+0000 7f69f674e700 -1 reset not still connected to 0x563077e0ca90 2021-07-10T09:33:10.981 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:10.976+0000 7f69f674e700 -1 reset not still connected to 0x563077e0d790 2021-07-10T09:33:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1386116096 bytes 2021-07-10T09:33:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1386723328 bytes 2021-07-10T09:33:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1404495872 bytes 2021-07-10T09:33:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1426413568 bytes 2021-07-10T09:33:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1503730688 bytes 2021-07-10T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1525103616 bytes 2021-07-10T09:33:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1638218752 bytes 2021-07-10T09:33:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1653221376 bytes 2021-07-10T09:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1721223168 bytes 2021-07-10T09:33:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1747138560 bytes 2021-07-10T09:33:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1788907520 bytes 2021-07-10T09:33:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1804263424 bytes 2021-07-10T09:33:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1845698560 bytes 2021-07-10T09:33:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1846310912 bytes 2021-07-10T09:33:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1849494528 bytes 2021-07-10T09:33:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:33:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:33:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:33:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:33:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:33:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_15 2021-07-10T09:33:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2021-07-10T09:33:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:33:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:33:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_15 128M 2021-07-10T09:33:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:33:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:33:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:33:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:33:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2021-07-10T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2021-07-10T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:33:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2021-07-10T09:33:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_15 2021-07-10T09:33:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:33:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:33:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=13 2021-07-10T09:33:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:33:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:33:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 67365888 bytes 2021-07-10T09:33:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 73362432 bytes 2021-07-10T09:33:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 100920320 bytes 2021-07-10T09:33:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 134568960 bytes 2021-07-10T09:33:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 183781376 bytes 2021-07-10T09:33:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 225247232 bytes 2021-07-10T09:33:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 259235840 bytes 2021-07-10T09:33:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 344963072 bytes 2021-07-10T09:33:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 358958080 bytes 2021-07-10T09:33:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 421743616 bytes 2021-07-10T09:33:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 466819072 bytes 2021-07-10T09:33:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 505482240 bytes 2021-07-10T09:33:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 559531008 bytes 2021-07-10T09:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 608440320 bytes 2021-07-10T09:33:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 631366656 bytes 2021-07-10T09:33:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 649993216 bytes 2021-07-10T09:33:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 650608640 bytes 2021-07-10T09:33:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 704570368 bytes 2021-07-10T09:33:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 725129216 bytes 2021-07-10T09:33:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 772724736 bytes 2021-07-10T09:33:22.999 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:33:22.991+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:33:22.999 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:33:22.992+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:33:22.999 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:33:22.992+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:33:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 836711424 bytes 2021-07-10T09:33:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 852865024 bytes 2021-07-10T09:33:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 876242944 bytes 2021-07-10T09:33:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 926994432 bytes 2021-07-10T09:33:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 951776256 bytes 2021-07-10T09:33:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1012699136 bytes 2021-07-10T09:33:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1118551040 bytes 2021-07-10T09:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1216001024 bytes 2021-07-10T09:33:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1232808960 bytes 2021-07-10T09:33:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1272427520 bytes 2021-07-10T09:33:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1301830656 bytes 2021-07-10T09:33:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1335490560 bytes 2021-07-10T09:33:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1383477248 bytes 2021-07-10T09:33:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1434319872 bytes 2021-07-10T09:33:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1453279232 bytes 2021-07-10T09:33:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1530903552 bytes 2021-07-10T09:33:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1584791552 bytes 2021-07-10T09:33:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1627501568 bytes 2021-07-10T09:33:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1643836416 bytes 2021-07-10T09:33:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:33:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:33:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:33:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:33:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:33:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_16 2021-07-10T09:33:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2021-07-10T09:33:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:33:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:33:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_16 128M 2021-07-10T09:33:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:33:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:33:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:33:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:33:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2021-07-10T09:33:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:33:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2021-07-10T09:33:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:33:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2021-07-10T09:33:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:33:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:33:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_16 2021-07-10T09:33:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:33:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:33:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:33:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:33:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:33:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 57136128 bytes 2021-07-10T09:33:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 105978880 bytes 2021-07-10T09:33:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 185814016 bytes 2021-07-10T09:33:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 208207872 bytes 2021-07-10T09:33:32.324 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:33:32.321+0000 7f25844e6700 -1 reset not still connected to 0x55574de83380 2021-07-10T09:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 271109120 bytes 2021-07-10T09:33:32.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 315226112 bytes 2021-07-10T09:33:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 358593536 bytes 2021-07-10T09:33:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 407831552 bytes 2021-07-10T09:33:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 437219328 bytes 2021-07-10T09:33:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 488665088 bytes 2021-07-10T09:33:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 538515456 bytes 2021-07-10T09:33:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 541907968 bytes 2021-07-10T09:33:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 599277568 bytes 2021-07-10T09:33:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 710341632 bytes 2021-07-10T09:33:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 735156224 bytes 2021-07-10T09:33:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 810835968 bytes 2021-07-10T09:33:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 834137088 bytes 2021-07-10T09:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 915996672 bytes 2021-07-10T09:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 1058599936 bytes 2021-07-10T09:33:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 1103677440 bytes 2021-07-10T09:33:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 1213543424 bytes 2021-07-10T09:33:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 1252905984 bytes 2021-07-10T09:33:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1306418176 bytes 2021-07-10T09:33:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1384952832 bytes 2021-07-10T09:33:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:33:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:33:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:33:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:33:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:33:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_17 2021-07-10T09:33:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2021-07-10T09:33:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:33:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:33:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_17 128M 2021-07-10T09:33:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:33:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:33:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:33:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:33:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2021-07-10T09:33:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:33:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2021-07-10T09:33:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2021-07-10T09:33:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:33:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:33:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_17 2021-07-10T09:33:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:33:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:33:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=11 2021-07-10T09:33:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:33:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:33:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 37853184 bytes 2021-07-10T09:33:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 79471616 bytes 2021-07-10T09:33:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 101497856 bytes 2021-07-10T09:33:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 139136000 bytes 2021-07-10T09:33:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 169032704 bytes 2021-07-10T09:33:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 202217472 bytes 2021-07-10T09:33:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 245068800 bytes 2021-07-10T09:33:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 285958144 bytes 2021-07-10T09:33:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 294893568 bytes 2021-07-10T09:33:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 314480640 bytes 2021-07-10T09:33:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 340032512 bytes 2021-07-10T09:33:45.428 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:33:45.426+0000 7f5115296700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:33:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 422857728 bytes 2021-07-10T09:33:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 453007360 bytes 2021-07-10T09:33:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 519392256 bytes 2021-07-10T09:33:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 605717504 bytes 2021-07-10T09:33:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 684595200 bytes 2021-07-10T09:33:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 710528000 bytes 2021-07-10T09:33:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 762819584 bytes 2021-07-10T09:33:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 845304832 bytes 2021-07-10T09:33:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 898723840 bytes 2021-07-10T09:33:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 910265344 bytes 2021-07-10T09:33:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 934610944 bytes 2021-07-10T09:33:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 941660160 bytes 2021-07-10T09:33:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1009031168 bytes 2021-07-10T09:33:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1048528896 bytes 2021-07-10T09:33:50.572 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:33:50.569+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:33:50.572 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:33:50.569+0000 7f511028c700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2021-07-10T09:33:50.572 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:33:50.569+0000 7f5110a8d700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:33:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1124002816 bytes 2021-07-10T09:33:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1161230336 bytes 2021-07-10T09:33:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1192913920 bytes 2021-07-10T09:33:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1249246208 bytes 2021-07-10T09:33:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1249833984 bytes 2021-07-10T09:33:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:33:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:33:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:33:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:33:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:33:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_18 2021-07-10T09:33:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2021-07-10T09:33:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:33:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:33:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_18 128M 2021-07-10T09:33:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:33:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:33:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:33:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:33:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:33:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2021-07-10T09:33:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:33:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2021-07-10T09:33:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2021-07-10T09:33:53.641 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:53.638+0000 7f69f674e700 -1 reset not still connected to 0x563077e0d790 2021-07-10T09:33:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:33:53.642 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:33:53.641+0000 7f69f674e700 -1 reset not still connected to 0x563077e0d790 2021-07-10T09:33:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:33:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_18 2021-07-10T09:33:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:33:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:33:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:33:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:33:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:33:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:33:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:33:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 8292352 bytes 2021-07-10T09:33:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 104143872 bytes 2021-07-10T09:33:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 125582336 bytes 2021-07-10T09:33:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 138245120 bytes 2021-07-10T09:33:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 188640256 bytes 2021-07-10T09:33:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 206232576 bytes 2021-07-10T09:33:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 278167552 bytes 2021-07-10T09:33:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 310468608 bytes 2021-07-10T09:33:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 350445568 bytes 2021-07-10T09:33:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 409733120 bytes 2021-07-10T09:33:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 523411456 bytes 2021-07-10T09:33:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 528045056 bytes 2021-07-10T09:33:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 559789056 bytes 2021-07-10T09:33:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 582184960 bytes 2021-07-10T09:33:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 619084800 bytes 2021-07-10T09:33:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 643228672 bytes 2021-07-10T09:33:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 722356224 bytes 2021-07-10T09:33:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 738022400 bytes 2021-07-10T09:34:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 775568384 bytes 2021-07-10T09:34:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 776172544 bytes 2021-07-10T09:34:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 818196480 bytes 2021-07-10T09:34:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 886854656 bytes 2021-07-10T09:34:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 959180800 bytes 2021-07-10T09:34:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1003707392 bytes 2021-07-10T09:34:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1066688512 bytes 2021-07-10T09:34:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1123953664 bytes 2021-07-10T09:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1172028416 bytes 2021-07-10T09:34:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1199862784 bytes 2021-07-10T09:34:03.646 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.638056703947607) > weight (0.25) 2021-07-10T09:34:03.646 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9429753156475432) > weight (0.25) 2021-07-10T09:34:03.646 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5184110498401747) > weight (0.25) 2021-07-10T09:34:03.646 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2021-07-10T09:34:03.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2021-07-10T09:34:03.648 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2021-07-10T09:34:03.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:2021-07-10T09:34:03.647+0000 7f7451a45700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 36889) UID: 1000 2021-07-10T09:34:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1239833600 bytes 2021-07-10T09:34:04.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:34:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1277466624 bytes 2021-07-10T09:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1341790208 bytes 2021-07-10T09:34:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1449860096 bytes 2021-07-10T09:34:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:34:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:34:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:34:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:34:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:34:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_19 2021-07-10T09:34:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2021-07-10T09:34:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:34:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:34:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_19 128M 2021-07-10T09:34:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:34:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:34:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:34:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:34:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2021-07-10T09:34:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:34:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2021-07-10T09:34:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:34:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2021-07-10T09:34:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:34:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:34:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_19 2021-07-10T09:34:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:34:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:34:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:34:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:34:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:34:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 46451712 bytes 2021-07-10T09:34:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 106735616 bytes 2021-07-10T09:34:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 134627328 bytes 2021-07-10T09:34:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 182556672 bytes 2021-07-10T09:34:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 225137664 bytes 2021-07-10T09:34:08.649 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2021-07-10T09:34:08.650 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:34:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:34:08.651 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2021-07-10T09:34:08.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2021-07-10T09:34:08.652 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2021-07-10T09:34:08.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2021-07-10T09:34:08.656 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2021-07-10T09:34:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 267777024 bytes 2021-07-10T09:34:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 294623232 bytes 2021-07-10T09:34:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 404426752 bytes 2021-07-10T09:34:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 456782848 bytes 2021-07-10T09:34:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 492233728 bytes 2021-07-10T09:34:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 567058432 bytes 2021-07-10T09:34:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 619272192 bytes 2021-07-10T09:34:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 640059392 bytes 2021-07-10T09:34:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:34:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:34:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:34:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:34:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:34:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_20 2021-07-10T09:34:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2021-07-10T09:34:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:34:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_20 128M 2021-07-10T09:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:34:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:34:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:34:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2021-07-10T09:34:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:34:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2021-07-10T09:34:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:34:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2021-07-10T09:34:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:34:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:34:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_20 2021-07-10T09:34:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:34:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:34:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:34:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:34:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:34:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 54234112 bytes 2021-07-10T09:34:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 106614784 bytes 2021-07-10T09:34:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 143670272 bytes 2021-07-10T09:34:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 164251648 bytes 2021-07-10T09:34:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 242607104 bytes 2021-07-10T09:34:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 253030400 bytes 2021-07-10T09:34:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 254003200 bytes 2021-07-10T09:34:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 287133696 bytes 2021-07-10T09:34:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 304898048 bytes 2021-07-10T09:34:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 320575488 bytes 2021-07-10T09:34:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 338616320 bytes 2021-07-10T09:34:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 382793728 bytes 2021-07-10T09:34:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 435079168 bytes 2021-07-10T09:34:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 487972864 bytes 2021-07-10T09:34:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 490607616 bytes 2021-07-10T09:34:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 541286400 bytes 2021-07-10T09:34:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 541886464 bytes 2021-07-10T09:34:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 593585152 bytes 2021-07-10T09:34:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 613898240 bytes 2021-07-10T09:34:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 753028096 bytes 2021-07-10T09:34:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 793545728 bytes 2021-07-10T09:34:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 895069184 bytes 2021-07-10T09:34:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 982845440 bytes 2021-07-10T09:34:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1043660800 bytes 2021-07-10T09:34:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1056434176 bytes 2021-07-10T09:34:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1102360576 bytes 2021-07-10T09:34:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1178521600 bytes 2021-07-10T09:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1194592256 bytes 2021-07-10T09:34:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1234835456 bytes 2021-07-10T09:34:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1264245760 bytes 2021-07-10T09:34:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1301722112 bytes 2021-07-10T09:34:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1357518848 bytes 2021-07-10T09:34:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1412952064 bytes 2021-07-10T09:34:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:34:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:34:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:34:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:34:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:34:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_21 2021-07-10T09:34:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2021-07-10T09:34:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:34:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:34:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_21 128M 2021-07-10T09:34:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:34:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:34:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:34:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:34:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2021-07-10T09:34:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:34:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2021-07-10T09:34:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:34:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2021-07-10T09:34:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:34:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:34:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_21 2021-07-10T09:34:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:34:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:34:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=9 2021-07-10T09:34:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:34:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:34:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 67511296 bytes 2021-07-10T09:34:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 87752704 bytes 2021-07-10T09:34:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 155617280 bytes 2021-07-10T09:34:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 191601664 bytes 2021-07-10T09:34:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 315463680 bytes 2021-07-10T09:34:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 335095808 bytes 2021-07-10T09:34:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 364890112 bytes 2021-07-10T09:34:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 413321216 bytes 2021-07-10T09:34:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 473552896 bytes 2021-07-10T09:34:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 495771648 bytes 2021-07-10T09:34:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 527961088 bytes 2021-07-10T09:34:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 577860608 bytes 2021-07-10T09:34:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 610918400 bytes 2021-07-10T09:34:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 628410368 bytes 2021-07-10T09:34:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 663291904 bytes 2021-07-10T09:34:31.428 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:34:31.426+0000 7f5115296700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:34:31.428 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:34:31.426+0000 7f511028c700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:34:31.429 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:34:31.426+0000 7f5114294700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:34:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 756324352 bytes 2021-07-10T09:34:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 818426880 bytes 2021-07-10T09:34:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 828023808 bytes 2021-07-10T09:34:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 854830080 bytes 2021-07-10T09:34:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 875326464 bytes 2021-07-10T09:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 894878720 bytes 2021-07-10T09:34:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 942702592 bytes 2021-07-10T09:34:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 994420736 bytes 2021-07-10T09:34:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1034440704 bytes 2021-07-10T09:34:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:34:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:34:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:34:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:34:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:34:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_22 2021-07-10T09:34:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2021-07-10T09:34:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:34:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:34:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_22 128M 2021-07-10T09:34:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:34:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:34:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:34:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:34:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2021-07-10T09:34:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:34:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2021-07-10T09:34:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:34:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2021-07-10T09:34:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:34:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_22 2021-07-10T09:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:34:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:34:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 10526720 bytes 2021-07-10T09:34:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 31200256 bytes 2021-07-10T09:34:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 37672960 bytes 2021-07-10T09:34:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 77109248 bytes 2021-07-10T09:34:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 119186432 bytes 2021-07-10T09:34:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 153937920 bytes 2021-07-10T09:34:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 205945856 bytes 2021-07-10T09:34:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 288268288 bytes 2021-07-10T09:34:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 311974912 bytes 2021-07-10T09:34:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 356399104 bytes 2021-07-10T09:34:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 404490240 bytes 2021-07-10T09:34:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 427187200 bytes 2021-07-10T09:34:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 454276096 bytes 2021-07-10T09:34:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 490654720 bytes 2021-07-10T09:34:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:34:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:34:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:34:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:34:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:34:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_23 2021-07-10T09:34:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2021-07-10T09:34:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:34:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:34:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_23 128M 2021-07-10T09:34:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:34:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:34:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:34:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:34:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2021-07-10T09:34:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:34:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2021-07-10T09:34:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:34:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2021-07-10T09:34:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:34:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:34:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_23 2021-07-10T09:34:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:34:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:34:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=11 2021-07-10T09:34:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:34:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:34:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 24313856 bytes 2021-07-10T09:34:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 61056000 bytes 2021-07-10T09:34:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 80617472 bytes 2021-07-10T09:34:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 82919424 bytes 2021-07-10T09:34:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 113337344 bytes 2021-07-10T09:34:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 159936512 bytes 2021-07-10T09:34:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 210064384 bytes 2021-07-10T09:34:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 257417216 bytes 2021-07-10T09:34:44.552 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2021-07-10T09:34:44.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2021-07-10T09:34:44.554 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2021-07-10T09:34:44.554 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2021-07-10T09:34:44.554 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:34:44.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi105.stderr:2021-07-10T09:34:44.554+0000 7f6671b6c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 46591) UID: 1000 2021-07-10T09:34:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 299556864 bytes 2021-07-10T09:34:44.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:34:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 343793664 bytes 2021-07-10T09:34:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 354534400 bytes 2021-07-10T09:34:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 384982016 bytes 2021-07-10T09:34:45.950 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:34:45.946+0000 7f51222b0700 -1 reset not still connected to 0x55a8e00d72b0 2021-07-10T09:34:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 411716608 bytes 2021-07-10T09:34:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 484130816 bytes 2021-07-10T09:34:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 506986496 bytes 2021-07-10T09:34:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 545378304 bytes 2021-07-10T09:34:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 599668736 bytes 2021-07-10T09:34:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 675186688 bytes 2021-07-10T09:34:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 675821568 bytes 2021-07-10T09:34:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 706940928 bytes 2021-07-10T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 735208448 bytes 2021-07-10T09:34:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 792041472 bytes 2021-07-10T09:34:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 833614848 bytes 2021-07-10T09:34:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 897010688 bytes 2021-07-10T09:34:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:34:50.666 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2021-07-10T09:34:50.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2021-07-10T09:34:50.667 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2021-07-10T09:34:50.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2021-07-10T09:34:50.718 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2021-07-10T09:34:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 920077312 bytes 2021-07-10T09:34:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 965662720 bytes 2021-07-10T09:34:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1020857344 bytes 2021-07-10T09:34:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1077376000 bytes 2021-07-10T09:34:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1136435200 bytes 2021-07-10T09:34:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:34:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:34:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:34:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:34:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:34:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_24 2021-07-10T09:34:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2021-07-10T09:34:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:34:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:34:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_24 128M 2021-07-10T09:34:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:34:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:34:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:34:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:34:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2021-07-10T09:34:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:34:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2021-07-10T09:34:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:34:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2021-07-10T09:34:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:34:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:34:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_24 2021-07-10T09:34:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:34:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:34:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:34:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:34:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:34:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 78584832 bytes 2021-07-10T09:34:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 116182016 bytes 2021-07-10T09:34:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 158615552 bytes 2021-07-10T09:34:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 165533696 bytes 2021-07-10T09:34:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 217081856 bytes 2021-07-10T09:34:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 286586880 bytes 2021-07-10T09:34:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 318660608 bytes 2021-07-10T09:34:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 339084288 bytes 2021-07-10T09:34:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 399365120 bytes 2021-07-10T09:34:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 399969280 bytes 2021-07-10T09:34:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 491229184 bytes 2021-07-10T09:34:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 491827200 bytes 2021-07-10T09:34:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 525760512 bytes 2021-07-10T09:34:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:34:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:34:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:34:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:34:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:34:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_25 2021-07-10T09:34:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2021-07-10T09:34:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:34:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:34:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_25 128M 2021-07-10T09:34:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:34:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:34:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:34:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:34:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:34:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2021-07-10T09:34:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:34:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2021-07-10T09:34:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:34:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:34:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2021-07-10T09:34:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:34:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:34:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_25 2021-07-10T09:34:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:34:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:34:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:34:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:34:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:34:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:34:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:35:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 133386240 bytes 2021-07-10T09:35:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 233355264 bytes 2021-07-10T09:35:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 243346432 bytes 2021-07-10T09:35:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 262958080 bytes 2021-07-10T09:35:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 297876480 bytes 2021-07-10T09:35:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 298469376 bytes 2021-07-10T09:35:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 334008320 bytes 2021-07-10T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 358780928 bytes 2021-07-10T09:35:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 359376896 bytes 2021-07-10T09:35:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 419503104 bytes 2021-07-10T09:35:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 443577344 bytes 2021-07-10T09:35:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 468504576 bytes 2021-07-10T09:35:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 502440960 bytes 2021-07-10T09:35:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 533629952 bytes 2021-07-10T09:35:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 561655808 bytes 2021-07-10T09:35:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 607429632 bytes 2021-07-10T09:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 665443328 bytes 2021-07-10T09:35:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 724891648 bytes 2021-07-10T09:35:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 733817856 bytes 2021-07-10T09:35:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 756758528 bytes 2021-07-10T09:35:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 781661184 bytes 2021-07-10T09:35:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:35:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:35:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:35:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:35:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:35:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_26 2021-07-10T09:35:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2021-07-10T09:35:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:35:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:35:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_26 128M 2021-07-10T09:35:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:35:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:35:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:35:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:35:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2021-07-10T09:35:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:35:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2021-07-10T09:35:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:35:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2021-07-10T09:35:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:35:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:35:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_26 2021-07-10T09:35:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:35:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:35:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=7 2021-07-10T09:35:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:35:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:35:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 35817472 bytes 2021-07-10T09:35:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 41531392 bytes 2021-07-10T09:35:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 57444352 bytes 2021-07-10T09:35:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 104548352 bytes 2021-07-10T09:35:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 127462400 bytes 2021-07-10T09:35:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 183853056 bytes 2021-07-10T09:35:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 222049280 bytes 2021-07-10T09:35:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 290256896 bytes 2021-07-10T09:35:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 298115072 bytes 2021-07-10T09:35:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 330057728 bytes 2021-07-10T09:35:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 445764608 bytes 2021-07-10T09:35:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 477029376 bytes 2021-07-10T09:35:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 501870592 bytes 2021-07-10T09:35:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 578110464 bytes 2021-07-10T09:35:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 678467584 bytes 2021-07-10T09:35:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 702901248 bytes 2021-07-10T09:35:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 751395840 bytes 2021-07-10T09:35:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:35:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:35:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:35:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:35:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:35:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_27 2021-07-10T09:35:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2021-07-10T09:35:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:35:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_27 128M 2021-07-10T09:35:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:35:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:35:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:35:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:35:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2021-07-10T09:35:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:35:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2021-07-10T09:35:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:35:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2021-07-10T09:35:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:35:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:35:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_27 2021-07-10T09:35:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:35:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:35:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:35:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:35:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:35:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 17040384 bytes 2021-07-10T09:35:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 51718144 bytes 2021-07-10T09:35:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 133596160 bytes 2021-07-10T09:35:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 182127616 bytes 2021-07-10T09:35:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 220241920 bytes 2021-07-10T09:35:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 255244288 bytes 2021-07-10T09:35:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 255823872 bytes 2021-07-10T09:35:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 327173120 bytes 2021-07-10T09:35:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 349871104 bytes 2021-07-10T09:35:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 405510144 bytes 2021-07-10T09:35:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 460219392 bytes 2021-07-10T09:35:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 529536000 bytes 2021-07-10T09:35:21.695 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:35:21.693+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 585058304 bytes 2021-07-10T09:35:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 634806272 bytes 2021-07-10T09:35:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 644143104 bytes 2021-07-10T09:35:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 666425344 bytes 2021-07-10T09:35:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 704240640 bytes 2021-07-10T09:35:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 715446272 bytes 2021-07-10T09:35:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 771355648 bytes 2021-07-10T09:35:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 815419392 bytes 2021-07-10T09:35:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 856413184 bytes 2021-07-10T09:35:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 920635392 bytes 2021-07-10T09:35:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 960440320 bytes 2021-07-10T09:35:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1013924864 bytes 2021-07-10T09:35:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1040235520 bytes 2021-07-10T09:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_28 2021-07-10T09:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2021-07-10T09:35:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:35:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:35:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_28 128M 2021-07-10T09:35:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:35:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:35:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:35:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:35:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2021-07-10T09:35:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:35:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2021-07-10T09:35:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:35:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2021-07-10T09:35:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:35:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:35:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_28 2021-07-10T09:35:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:35:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:35:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:35:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:35:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:35:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 49228800 bytes 2021-07-10T09:35:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 74243072 bytes 2021-07-10T09:35:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 110486528 bytes 2021-07-10T09:35:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 156905472 bytes 2021-07-10T09:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 193655808 bytes 2021-07-10T09:35:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 230667264 bytes 2021-07-10T09:35:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 304621568 bytes 2021-07-10T09:35:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 320681984 bytes 2021-07-10T09:35:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 353730560 bytes 2021-07-10T09:35:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 438659072 bytes 2021-07-10T09:35:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 465893376 bytes 2021-07-10T09:35:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 498423808 bytes 2021-07-10T09:35:32.657 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2021-07-10T09:35:32.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2021-07-10T09:35:32.658 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2021-07-10T09:35:32.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:35:32.657+0000 7f6fa3c70700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48462) UID: 1000 2021-07-10T09:35:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 533226496 bytes 2021-07-10T09:35:33.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:35:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 582625280 bytes 2021-07-10T09:35:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 627140608 bytes 2021-07-10T09:35:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 645840896 bytes 2021-07-10T09:35:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 727890944 bytes 2021-07-10T09:35:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 756400128 bytes 2021-07-10T09:35:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 783045632 bytes 2021-07-10T09:35:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 805923840 bytes 2021-07-10T09:35:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 834581504 bytes 2021-07-10T09:35:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 849114112 bytes 2021-07-10T09:35:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 852498432 bytes 2021-07-10T09:35:36.662 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2021-07-10T09:35:36.663 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:35:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:35:36.664 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2021-07-10T09:35:36.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:35:36.664 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:35:36.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:35:36.677 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2021-07-10T09:35:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 864652288 bytes 2021-07-10T09:35:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 928231424 bytes 2021-07-10T09:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:35:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:35:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:35:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:35:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_29 2021-07-10T09:35:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2021-07-10T09:35:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:35:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:35:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_29 128M 2021-07-10T09:35:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:35:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:35:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:35:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:35:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2021-07-10T09:35:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:35:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2021-07-10T09:35:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:35:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2021-07-10T09:35:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:35:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:35:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_29 2021-07-10T09:35:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:35:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:35:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=7 2021-07-10T09:35:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:35:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:35:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 30049280 bytes 2021-07-10T09:35:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 34607104 bytes 2021-07-10T09:35:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 66558976 bytes 2021-07-10T09:35:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 110423040 bytes 2021-07-10T09:35:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 181924864 bytes 2021-07-10T09:35:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 236718080 bytes 2021-07-10T09:35:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 304934912 bytes 2021-07-10T09:35:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 305558528 bytes 2021-07-10T09:35:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 360246272 bytes 2021-07-10T09:35:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 418006016 bytes 2021-07-10T09:35:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 473133056 bytes 2021-07-10T09:35:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 553085952 bytes 2021-07-10T09:35:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 641462272 bytes 2021-07-10T09:35:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 679528448 bytes 2021-07-10T09:35:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 729607168 bytes 2021-07-10T09:35:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 744509440 bytes 2021-07-10T09:35:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 827596800 bytes 2021-07-10T09:35:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 882338816 bytes 2021-07-10T09:35:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:35:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:35:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:35:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:35:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:35:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_30 2021-07-10T09:35:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2021-07-10T09:35:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:35:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:35:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_30 128M 2021-07-10T09:35:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:35:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:35:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:35:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:35:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2021-07-10T09:35:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:35:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2021-07-10T09:35:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:35:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2021-07-10T09:35:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:35:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:35:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_30 2021-07-10T09:35:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:35:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:35:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=6 2021-07-10T09:35:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:35:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:35:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 32864256 bytes 2021-07-10T09:35:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 44862464 bytes 2021-07-10T09:35:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 64904192 bytes 2021-07-10T09:35:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 135510016 bytes 2021-07-10T09:35:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 209666048 bytes 2021-07-10T09:35:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 224586752 bytes 2021-07-10T09:35:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 282028032 bytes 2021-07-10T09:35:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 385472512 bytes 2021-07-10T09:35:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 446629888 bytes 2021-07-10T09:35:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 525281280 bytes 2021-07-10T09:35:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 556680192 bytes 2021-07-10T09:35:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 614348800 bytes 2021-07-10T09:35:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 623963136 bytes 2021-07-10T09:35:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 627666944 bytes 2021-07-10T09:35:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 688919552 bytes 2021-07-10T09:35:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 689538048 bytes 2021-07-10T09:35:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 718902272 bytes 2021-07-10T09:35:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:35:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:35:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:35:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:35:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:35:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_31 2021-07-10T09:35:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2021-07-10T09:35:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:35:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:35:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_31 128M 2021-07-10T09:35:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:35:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:35:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:35:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:35:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2021-07-10T09:35:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:35:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2021-07-10T09:35:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:35:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2021-07-10T09:35:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:35:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:35:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_31 2021-07-10T09:35:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:35:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:35:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=6 2021-07-10T09:35:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:35:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:35:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 21279744 bytes 2021-07-10T09:35:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 53022720 bytes 2021-07-10T09:35:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 87905280 bytes 2021-07-10T09:35:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 152855552 bytes 2021-07-10T09:35:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 207266816 bytes 2021-07-10T09:35:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 300095488 bytes 2021-07-10T09:35:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 315680768 bytes 2021-07-10T09:35:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 350966784 bytes 2021-07-10T09:35:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 385236992 bytes 2021-07-10T09:35:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 412792832 bytes 2021-07-10T09:35:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 487895040 bytes 2021-07-10T09:35:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 551280640 bytes 2021-07-10T09:35:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 594249728 bytes 2021-07-10T09:35:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 599874560 bytes 2021-07-10T09:35:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 626589696 bytes 2021-07-10T09:35:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 667379712 bytes 2021-07-10T09:35:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 681123840 bytes 2021-07-10T09:35:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:35:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:35:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:35:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:35:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:35:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_32 2021-07-10T09:35:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2021-07-10T09:35:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:35:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:35:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_32 128M 2021-07-10T09:35:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:35:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:35:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:35:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:35:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:35:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2021-07-10T09:35:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:35:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2021-07-10T09:35:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:35:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:35:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2021-07-10T09:36:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:36:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:36:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_32 2021-07-10T09:36:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:36:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:36:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=7 2021-07-10T09:36:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:36:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:36:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 46967808 bytes 2021-07-10T09:36:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 99511296 bytes 2021-07-10T09:36:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 144675840 bytes 2021-07-10T09:36:02.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 193866752 bytes 2021-07-10T09:36:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 264838144 bytes 2021-07-10T09:36:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 289581056 bytes 2021-07-10T09:36:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 325927936 bytes 2021-07-10T09:36:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 369695744 bytes 2021-07-10T09:36:03.833 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:36:03.830+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:36:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 406633472 bytes 2021-07-10T09:36:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 459709440 bytes 2021-07-10T09:36:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 509961216 bytes 2021-07-10T09:36:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 545462272 bytes 2021-07-10T09:36:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 571392000 bytes 2021-07-10T09:36:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 639693824 bytes 2021-07-10T09:36:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 706449408 bytes 2021-07-10T09:36:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 751895552 bytes 2021-07-10T09:36:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 829106176 bytes 2021-07-10T09:36:06.871 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:36:06.869+0000 7f69f674e700 -1 reset not still connected to 0x563073ea7ee0 2021-07-10T09:36:06.871 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:36:06.869+0000 7f69f674e700 -1 reset not still connected to 0x5630859ba750 2021-07-10T09:36:06.872 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:36:06.869+0000 7f69f674e700 -1 reset not still connected to 0x56309b4bca90 2021-07-10T09:36:06.872 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:36:06.869+0000 7f69f674e700 -1 reset not still connected to 0x5630d8bdbee0 2021-07-10T09:36:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 838489088 bytes 2021-07-10T09:36:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 888247296 bytes 2021-07-10T09:36:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:36:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:36:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:36:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:36:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:36:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_33 2021-07-10T09:36:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2021-07-10T09:36:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:36:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:36:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_33 128M 2021-07-10T09:36:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:36:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:36:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:36:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:36:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2021-07-10T09:36:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:36:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2021-07-10T09:36:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:36:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2021-07-10T09:36:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:36:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:36:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_33 2021-07-10T09:36:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:36:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:36:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:36:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:36:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:36:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 51348480 bytes 2021-07-10T09:36:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 72796160 bytes 2021-07-10T09:36:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 155489280 bytes 2021-07-10T09:36:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 202512384 bytes 2021-07-10T09:36:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 270272512 bytes 2021-07-10T09:36:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 311077888 bytes 2021-07-10T09:36:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 363057152 bytes 2021-07-10T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 396475392 bytes 2021-07-10T09:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 420135936 bytes 2021-07-10T09:36:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 448801792 bytes 2021-07-10T09:36:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 501797888 bytes 2021-07-10T09:36:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 521825280 bytes 2021-07-10T09:36:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 551547904 bytes 2021-07-10T09:36:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 589520896 bytes 2021-07-10T09:36:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 661069824 bytes 2021-07-10T09:36:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 698504192 bytes 2021-07-10T09:36:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 699780096 bytes 2021-07-10T09:36:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 700389376 bytes 2021-07-10T09:36:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 721269760 bytes 2021-07-10T09:36:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 736429056 bytes 2021-07-10T09:36:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 828985344 bytes 2021-07-10T09:36:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 848423936 bytes 2021-07-10T09:36:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 871244800 bytes 2021-07-10T09:36:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 913504256 bytes 2021-07-10T09:36:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 937451520 bytes 2021-07-10T09:36:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 938061824 bytes 2021-07-10T09:36:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 973323264 bytes 2021-07-10T09:36:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 987552768 bytes 2021-07-10T09:36:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1057440768 bytes 2021-07-10T09:36:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1104524288 bytes 2021-07-10T09:36:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1169977344 bytes 2021-07-10T09:36:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1221623808 bytes 2021-07-10T09:36:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1244614656 bytes 2021-07-10T09:36:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1275574272 bytes 2021-07-10T09:36:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:36:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:36:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:36:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:36:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:36:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_34 2021-07-10T09:36:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2021-07-10T09:36:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:36:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:36:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_34 128M 2021-07-10T09:36:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:36:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:36:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:36:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:36:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2021-07-10T09:36:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:36:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2021-07-10T09:36:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:36:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2021-07-10T09:36:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:36:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:36:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_34 2021-07-10T09:36:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:36:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:36:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=9 2021-07-10T09:36:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:36:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:36:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 34136064 bytes 2021-07-10T09:36:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 39908352 bytes 2021-07-10T09:36:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 55029760 bytes 2021-07-10T09:36:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 85228544 bytes 2021-07-10T09:36:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 151599104 bytes 2021-07-10T09:36:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 205983744 bytes 2021-07-10T09:36:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 230618112 bytes 2021-07-10T09:36:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 263344128 bytes 2021-07-10T09:36:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 292930560 bytes 2021-07-10T09:36:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 312345600 bytes 2021-07-10T09:36:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 389809152 bytes 2021-07-10T09:36:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 444943360 bytes 2021-07-10T09:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 489229312 bytes 2021-07-10T09:36:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 513409024 bytes 2021-07-10T09:36:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 535120896 bytes 2021-07-10T09:36:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 582896640 bytes 2021-07-10T09:36:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 607283200 bytes 2021-07-10T09:36:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 612250624 bytes 2021-07-10T09:36:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 666668032 bytes 2021-07-10T09:36:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 706356224 bytes 2021-07-10T09:36:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 727024640 bytes 2021-07-10T09:36:29.291 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:36:29.288+0000 7f25844e6700 -1 reset not still connected to 0x55576738f5f0 2021-07-10T09:36:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 766735360 bytes 2021-07-10T09:36:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 828777472 bytes 2021-07-10T09:36:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 903581696 bytes 2021-07-10T09:36:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:36:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:36:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:36:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:36:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:36:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_35 2021-07-10T09:36:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2021-07-10T09:36:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:36:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:36:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_35 128M 2021-07-10T09:36:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:36:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:36:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:36:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:36:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2021-07-10T09:36:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:36:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2021-07-10T09:36:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:36:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2021-07-10T09:36:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:36:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:36:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_35 2021-07-10T09:36:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:36:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:36:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=11 2021-07-10T09:36:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:36:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:36:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 21923840 bytes 2021-07-10T09:36:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 64718848 bytes 2021-07-10T09:36:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 88919040 bytes 2021-07-10T09:36:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 146443264 bytes 2021-07-10T09:36:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 154796032 bytes 2021-07-10T09:36:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 169581568 bytes 2021-07-10T09:36:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 219143168 bytes 2021-07-10T09:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 270373888 bytes 2021-07-10T09:36:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 303894528 bytes 2021-07-10T09:36:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 323325952 bytes 2021-07-10T09:36:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 345359360 bytes 2021-07-10T09:36:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 361690112 bytes 2021-07-10T09:36:35.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9452892026878073) > weight (0.25) 2021-07-10T09:36:35.720 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9931782536942128) > weight (0.25) 2021-07-10T09:36:35.720 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4587350027536984) > weight (0.25) 2021-07-10T09:36:35.720 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3277740657527043) > weight (0.25) 2021-07-10T09:36:35.720 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2021-07-10T09:36:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 406808576 bytes 2021-07-10T09:36:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 464407552 bytes 2021-07-10T09:36:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 504296448 bytes 2021-07-10T09:36:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 541599744 bytes 2021-07-10T09:36:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 567273472 bytes 2021-07-10T09:36:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 583341056 bytes 2021-07-10T09:36:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 649706496 bytes 2021-07-10T09:36:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 730980352 bytes 2021-07-10T09:36:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 779125760 bytes 2021-07-10T09:36:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 845891584 bytes 2021-07-10T09:36:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 889728000 bytes 2021-07-10T09:36:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 890359808 bytes 2021-07-10T09:36:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 906547200 bytes 2021-07-10T09:36:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 916182016 bytes 2021-07-10T09:36:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 969105408 bytes 2021-07-10T09:36:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 991959040 bytes 2021-07-10T09:36:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1020232704 bytes 2021-07-10T09:36:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1032145920 bytes 2021-07-10T09:36:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1109959680 bytes 2021-07-10T09:36:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:36:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:36:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:36:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:36:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:36:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_36 2021-07-10T09:36:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2021-07-10T09:36:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:36:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_36 128M 2021-07-10T09:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:36:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:36:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:36:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:36:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2021-07-10T09:36:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:36:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2021-07-10T09:36:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:36:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2021-07-10T09:36:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:36:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:36:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_36 2021-07-10T09:36:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:36:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:36:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:36:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:36:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:36:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 2961408 bytes 2021-07-10T09:36:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 22112256 bytes 2021-07-10T09:36:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 85122048 bytes 2021-07-10T09:36:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 164078592 bytes 2021-07-10T09:36:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 220318720 bytes 2021-07-10T09:36:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 251018240 bytes 2021-07-10T09:36:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 251638784 bytes 2021-07-10T09:36:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 355273728 bytes 2021-07-10T09:36:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 393309184 bytes 2021-07-10T09:36:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 455372800 bytes 2021-07-10T09:36:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 462314496 bytes 2021-07-10T09:36:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 474487808 bytes 2021-07-10T09:36:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 546096128 bytes 2021-07-10T09:36:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:36:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:36:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:36:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:36:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:36:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_37 2021-07-10T09:36:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2021-07-10T09:36:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:36:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:36:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_37 128M 2021-07-10T09:36:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:36:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:36:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:36:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:36:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2021-07-10T09:36:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:36:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2021-07-10T09:36:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:36:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2021-07-10T09:36:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:36:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:36:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_37 2021-07-10T09:36:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:36:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:36:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:36:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:36:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:36:49.929 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:36:49.927+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2021-07-10T09:36:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 85747712 bytes 2021-07-10T09:36:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 133071872 bytes 2021-07-10T09:36:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 170977280 bytes 2021-07-10T09:36:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 226039808 bytes 2021-07-10T09:36:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 285261824 bytes 2021-07-10T09:36:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 311203840 bytes 2021-07-10T09:36:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 406942720 bytes 2021-07-10T09:36:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 488931328 bytes 2021-07-10T09:36:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 525146112 bytes 2021-07-10T09:36:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 599575552 bytes 2021-07-10T09:36:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 643389440 bytes 2021-07-10T09:36:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 665067520 bytes 2021-07-10T09:36:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 704790528 bytes 2021-07-10T09:36:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 805320704 bytes 2021-07-10T09:36:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 866893824 bytes 2021-07-10T09:36:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 910099456 bytes 2021-07-10T09:36:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 946067456 bytes 2021-07-10T09:36:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 1044274176 bytes 2021-07-10T09:36:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 1069769728 bytes 2021-07-10T09:36:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 1115348992 bytes 2021-07-10T09:36:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:36:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:36:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_38 2021-07-10T09:36:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2021-07-10T09:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:36:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:36:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_38 128M 2021-07-10T09:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:36:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:36:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:36:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:36:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:36:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2021-07-10T09:36:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:36:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2021-07-10T09:36:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:36:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:36:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2021-07-10T09:36:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:36:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:36:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_38 2021-07-10T09:36:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:36:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:36:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:36:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:36:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=14 2021-07-10T09:36:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:36:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:36:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 36126720 bytes 2021-07-10T09:36:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 92025856 bytes 2021-07-10T09:36:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 101041152 bytes 2021-07-10T09:37:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 163700736 bytes 2021-07-10T09:37:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 215340032 bytes 2021-07-10T09:37:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 266873856 bytes 2021-07-10T09:37:01.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 286125056 bytes 2021-07-10T09:37:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 349172736 bytes 2021-07-10T09:37:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 401632256 bytes 2021-07-10T09:37:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 408970240 bytes 2021-07-10T09:37:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 475600896 bytes 2021-07-10T09:37:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 542525440 bytes 2021-07-10T09:37:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 586347520 bytes 2021-07-10T09:37:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 631472128 bytes 2021-07-10T09:37:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 722581504 bytes 2021-07-10T09:37:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 727105536 bytes 2021-07-10T09:37:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 737997824 bytes 2021-07-10T09:37:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 786045952 bytes 2021-07-10T09:37:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 848537600 bytes 2021-07-10T09:37:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 857977856 bytes 2021-07-10T09:37:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 880230400 bytes 2021-07-10T09:37:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 924564480 bytes 2021-07-10T09:37:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 958861312 bytes 2021-07-10T09:37:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 974086144 bytes 2021-07-10T09:37:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1014352896 bytes 2021-07-10T09:37:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1049679872 bytes 2021-07-10T09:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1056708608 bytes 2021-07-10T09:37:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1091150848 bytes 2021-07-10T09:37:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1166760960 bytes 2021-07-10T09:37:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1206456320 bytes 2021-07-10T09:37:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1285982208 bytes 2021-07-10T09:37:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1357201408 bytes 2021-07-10T09:37:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1394193408 bytes 2021-07-10T09:37:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1396698112 bytes 2021-07-10T09:37:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1470425088 bytes 2021-07-10T09:37:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1486759936 bytes 2021-07-10T09:37:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1512851456 bytes 2021-07-10T09:37:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1529033728 bytes 2021-07-10T09:37:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1575334912 bytes 2021-07-10T09:37:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1615895552 bytes 2021-07-10T09:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1654738944 bytes 2021-07-10T09:37:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:37:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:37:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:37:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:37:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:37:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_39 2021-07-10T09:37:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2021-07-10T09:37:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:37:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:37:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_39 128M 2021-07-10T09:37:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:37:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:37:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:37:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:37:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2021-07-10T09:37:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:37:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2021-07-10T09:37:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:37:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2021-07-10T09:37:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:37:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:37:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_39 2021-07-10T09:37:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:37:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:37:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=13 2021-07-10T09:37:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:37:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:37:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 616448 bytes 2021-07-10T09:37:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 45987840 bytes 2021-07-10T09:37:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 77634560 bytes 2021-07-10T09:37:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 147927040 bytes 2021-07-10T09:37:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 157025280 bytes 2021-07-10T09:37:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 198617088 bytes 2021-07-10T09:37:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 262266880 bytes 2021-07-10T09:37:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 288075776 bytes 2021-07-10T09:37:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 311798784 bytes 2021-07-10T09:37:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 348034048 bytes 2021-07-10T09:37:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 379554816 bytes 2021-07-10T09:37:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 438887424 bytes 2021-07-10T09:37:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 502940672 bytes 2021-07-10T09:37:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 565902336 bytes 2021-07-10T09:37:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 626310144 bytes 2021-07-10T09:37:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 701260800 bytes 2021-07-10T09:37:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 702830592 bytes 2021-07-10T09:37:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 758425600 bytes 2021-07-10T09:37:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 802156544 bytes 2021-07-10T09:37:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 849639424 bytes 2021-07-10T09:37:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 850269184 bytes 2021-07-10T09:37:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 881072128 bytes 2021-07-10T09:37:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 910117888 bytes 2021-07-10T09:37:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 947992576 bytes 2021-07-10T09:37:19.993 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:37:19.989+0000 7f25794d0700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:37:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1032931328 bytes 2021-07-10T09:37:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1047134208 bytes 2021-07-10T09:37:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1067429888 bytes 2021-07-10T09:37:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1112501248 bytes 2021-07-10T09:37:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1146376192 bytes 2021-07-10T09:37:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1200562176 bytes 2021-07-10T09:37:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1216216064 bytes 2021-07-10T09:37:21.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1251935232 bytes 2021-07-10T09:37:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1362213888 bytes 2021-07-10T09:37:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1416650752 bytes 2021-07-10T09:37:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1445499904 bytes 2021-07-10T09:37:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1468141568 bytes 2021-07-10T09:37:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1494200320 bytes 2021-07-10T09:37:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1534580736 bytes 2021-07-10T09:37:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1638257664 bytes 2021-07-10T09:37:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1715500032 bytes 2021-07-10T09:37:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1763238912 bytes 2021-07-10T09:37:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1792540672 bytes 2021-07-10T09:37:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1843038208 bytes 2021-07-10T09:37:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1871932416 bytes 2021-07-10T09:37:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1896494080 bytes 2021-07-10T09:37:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 1959420928 bytes 2021-07-10T09:37:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 1967425536 bytes 2021-07-10T09:37:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4800 IOs, 2006004736 bytes 2021-07-10T09:37:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4900 IOs, 2019516416 bytes 2021-07-10T09:37:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5000 IOs, 2093735936 bytes 2021-07-10T09:37:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5100 IOs, 2160094208 bytes 2021-07-10T09:37:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5200 IOs, 2207540224 bytes 2021-07-10T09:37:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:37:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:37:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:37:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:37:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:37:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_40 2021-07-10T09:37:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2021-07-10T09:37:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:37:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:37:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_40 128M 2021-07-10T09:37:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:37:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:37:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:37:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:37:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2021-07-10T09:37:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:37:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2021-07-10T09:37:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:37:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2021-07-10T09:37:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_40 2021-07-10T09:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:37:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=11 2021-07-10T09:37:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:37:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:37:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 44891136 bytes 2021-07-10T09:37:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 56554496 bytes 2021-07-10T09:37:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 62494720 bytes 2021-07-10T09:37:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 68258816 bytes 2021-07-10T09:37:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 142550016 bytes 2021-07-10T09:37:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 172672000 bytes 2021-07-10T09:37:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 187622400 bytes 2021-07-10T09:37:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 231428096 bytes 2021-07-10T09:37:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 297419776 bytes 2021-07-10T09:37:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 336109568 bytes 2021-07-10T09:37:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 368285696 bytes 2021-07-10T09:37:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 400494592 bytes 2021-07-10T09:37:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 463583232 bytes 2021-07-10T09:37:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 537498624 bytes 2021-07-10T09:37:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 564926464 bytes 2021-07-10T09:37:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 601396224 bytes 2021-07-10T09:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 627352576 bytes 2021-07-10T09:37:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 668774400 bytes 2021-07-10T09:37:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 721129472 bytes 2021-07-10T09:37:32.178 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:37:32.177+0000 7f5116298700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 5 2021-07-10T09:37:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 755221504 bytes 2021-07-10T09:37:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 801364992 bytes 2021-07-10T09:37:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 826301440 bytes 2021-07-10T09:37:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 854676480 bytes 2021-07-10T09:37:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 944093184 bytes 2021-07-10T09:37:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 970047488 bytes 2021-07-10T09:37:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1006485504 bytes 2021-07-10T09:37:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1007105024 bytes 2021-07-10T09:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1052568576 bytes 2021-07-10T09:37:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1074534400 bytes 2021-07-10T09:37:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1102217216 bytes 2021-07-10T09:37:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1146447872 bytes 2021-07-10T09:37:34.678 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9277619926719199) > weight (0.25) 2021-07-10T09:37:34.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.45669059546835244) > weight (0.25) 2021-07-10T09:37:34.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6305872637207488) > weight (0.25) 2021-07-10T09:37:34.679 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2021-07-10T09:37:34.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2021-07-10T09:37:34.680 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2021-07-10T09:37:34.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:2021-07-10T09:37:34.680+0000 7f6cb1e08700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 48993) UID: 1000 2021-07-10T09:37:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1191571456 bytes 2021-07-10T09:37:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1217680384 bytes 2021-07-10T09:37:35.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:37:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1228281856 bytes 2021-07-10T09:37:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1281382400 bytes 2021-07-10T09:37:35.682 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2021-07-10T09:37:35.682 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:37:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:37:35.683 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2021-07-10T09:37:35.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2021-07-10T09:37:35.684 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2021-07-10T09:37:35.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2021-07-10T09:37:35.692 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2021-07-10T09:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1328460800 bytes 2021-07-10T09:37:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1348835328 bytes 2021-07-10T09:37:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1386855424 bytes 2021-07-10T09:37:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1409482752 bytes 2021-07-10T09:37:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1466766336 bytes 2021-07-10T09:37:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1493920768 bytes 2021-07-10T09:37:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1550510080 bytes 2021-07-10T09:37:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1599808512 bytes 2021-07-10T09:37:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1610911744 bytes 2021-07-10T09:37:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1687113728 bytes 2021-07-10T09:37:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 1730475008 bytes 2021-07-10T09:37:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 1787743232 bytes 2021-07-10T09:37:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:37:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:37:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:37:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:37:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:37:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_41 2021-07-10T09:37:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2021-07-10T09:37:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:37:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:37:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_41 128M 2021-07-10T09:37:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:37:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:37:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:37:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:37:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2021-07-10T09:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2021-07-10T09:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:37:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2021-07-10T09:37:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:37:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_41 2021-07-10T09:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:37:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:37:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:37:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 81717248 bytes 2021-07-10T09:37:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 117958656 bytes 2021-07-10T09:37:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 166052864 bytes 2021-07-10T09:37:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 231810048 bytes 2021-07-10T09:37:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 275796992 bytes 2021-07-10T09:37:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 390814720 bytes 2021-07-10T09:37:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 427184128 bytes 2021-07-10T09:37:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 460661760 bytes 2021-07-10T09:37:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 488048640 bytes 2021-07-10T09:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 501791744 bytes 2021-07-10T09:37:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 538549248 bytes 2021-07-10T09:37:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 556559360 bytes 2021-07-10T09:37:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 617428992 bytes 2021-07-10T09:37:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 672992256 bytes 2021-07-10T09:37:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 683557888 bytes 2021-07-10T09:37:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 684158976 bytes 2021-07-10T09:37:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 729177088 bytes 2021-07-10T09:37:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 772358144 bytes 2021-07-10T09:37:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 816868352 bytes 2021-07-10T09:37:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 824695808 bytes 2021-07-10T09:37:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:37:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:37:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:37:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:37:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:37:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_42 2021-07-10T09:37:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2021-07-10T09:37:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:37:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:37:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_42 128M 2021-07-10T09:37:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:37:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:37:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:37:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:37:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2021-07-10T09:37:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:37:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2021-07-10T09:37:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:37:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2021-07-10T09:37:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:37:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:37:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_42 2021-07-10T09:37:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:37:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:37:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:37:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:37:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:37:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 69010432 bytes 2021-07-10T09:37:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 94417920 bytes 2021-07-10T09:37:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 132672512 bytes 2021-07-10T09:37:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 184065024 bytes 2021-07-10T09:37:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 208149504 bytes 2021-07-10T09:37:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 270780416 bytes 2021-07-10T09:37:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 306199552 bytes 2021-07-10T09:37:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 383995904 bytes 2021-07-10T09:37:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 389846016 bytes 2021-07-10T09:37:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 466256896 bytes 2021-07-10T09:37:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 562380800 bytes 2021-07-10T09:37:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 572534784 bytes 2021-07-10T09:37:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 653058048 bytes 2021-07-10T09:37:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 702531584 bytes 2021-07-10T09:37:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 741211136 bytes 2021-07-10T09:37:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 820126720 bytes 2021-07-10T09:37:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 838493184 bytes 2021-07-10T09:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 884451328 bytes 2021-07-10T09:37:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 943897600 bytes 2021-07-10T09:37:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 1007012864 bytes 2021-07-10T09:37:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 1043746816 bytes 2021-07-10T09:37:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 1103645696 bytes 2021-07-10T09:37:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1180918784 bytes 2021-07-10T09:37:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1212680192 bytes 2021-07-10T09:37:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1269094400 bytes 2021-07-10T09:37:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1270112256 bytes 2021-07-10T09:37:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1290690560 bytes 2021-07-10T09:37:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1327828992 bytes 2021-07-10T09:37:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1417340928 bytes 2021-07-10T09:37:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1517773824 bytes 2021-07-10T09:37:53.839 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:37:53.837+0000 7f25844e6700 -1 reset not still connected to 0x555754b0bba0 2021-07-10T09:37:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1532067840 bytes 2021-07-10T09:37:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1554314240 bytes 2021-07-10T09:37:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1582683136 bytes 2021-07-10T09:37:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1637671936 bytes 2021-07-10T09:37:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1683795968 bytes 2021-07-10T09:37:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1740359680 bytes 2021-07-10T09:37:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1775595520 bytes 2021-07-10T09:37:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1818452992 bytes 2021-07-10T09:37:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1819074560 bytes 2021-07-10T09:37:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1888009216 bytes 2021-07-10T09:37:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1911405568 bytes 2021-07-10T09:37:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1940840448 bytes 2021-07-10T09:37:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1976185856 bytes 2021-07-10T09:37:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 2002161664 bytes 2021-07-10T09:37:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:37:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:37:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:37:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:37:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:37:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_43 2021-07-10T09:37:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2021-07-10T09:37:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:37:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:37:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_43 128M 2021-07-10T09:37:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:37:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:37:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:37:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:37:57.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:37:57.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2021-07-10T09:37:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:37:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2021-07-10T09:37:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:37:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:37:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2021-07-10T09:37:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:37:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:37:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_43 2021-07-10T09:37:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:37:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:37:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:37:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:37:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=6 2021-07-10T09:37:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:37:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:37:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 32411648 bytes 2021-07-10T09:37:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 86397952 bytes 2021-07-10T09:37:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 200835072 bytes 2021-07-10T09:37:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 217884672 bytes 2021-07-10T09:37:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 231593984 bytes 2021-07-10T09:37:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 294064128 bytes 2021-07-10T09:37:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 362192896 bytes 2021-07-10T09:37:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 386039808 bytes 2021-07-10T09:37:59.992 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:37:59.990+0000 7f5113a93700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:38:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 425874432 bytes 2021-07-10T09:38:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 450283520 bytes 2021-07-10T09:38:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 473864192 bytes 2021-07-10T09:38:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 486106112 bytes 2021-07-10T09:38:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 506757120 bytes 2021-07-10T09:38:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 576722944 bytes 2021-07-10T09:38:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 628340736 bytes 2021-07-10T09:38:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 660275200 bytes 2021-07-10T09:38:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 706258944 bytes 2021-07-10T09:38:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 794579968 bytes 2021-07-10T09:38:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 850884608 bytes 2021-07-10T09:38:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 874464256 bytes 2021-07-10T09:38:03.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 879062016 bytes 2021-07-10T09:38:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 915627008 bytes 2021-07-10T09:38:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 957007872 bytes 2021-07-10T09:38:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:38:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:38:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:38:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:38:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:38:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_44 2021-07-10T09:38:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2021-07-10T09:38:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:38:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:38:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_44 128M 2021-07-10T09:38:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:38:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:38:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:38:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:38:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2021-07-10T09:38:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:38:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2021-07-10T09:38:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:38:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2021-07-10T09:38:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:38:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:38:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_44 2021-07-10T09:38:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:38:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:38:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:38:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:38:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:38:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 42349568 bytes 2021-07-10T09:38:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 71683072 bytes 2021-07-10T09:38:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 169779200 bytes 2021-07-10T09:38:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 252032000 bytes 2021-07-10T09:38:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 285076480 bytes 2021-07-10T09:38:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 317286400 bytes 2021-07-10T09:38:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 337417216 bytes 2021-07-10T09:38:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 374734848 bytes 2021-07-10T09:38:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 418163712 bytes 2021-07-10T09:38:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 502111232 bytes 2021-07-10T09:38:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 528147456 bytes 2021-07-10T09:38:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 546541568 bytes 2021-07-10T09:38:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 580561920 bytes 2021-07-10T09:38:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 620561408 bytes 2021-07-10T09:38:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 674563072 bytes 2021-07-10T09:38:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 720748544 bytes 2021-07-10T09:38:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 759811072 bytes 2021-07-10T09:38:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 769056768 bytes 2021-07-10T09:38:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 831915008 bytes 2021-07-10T09:38:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:38:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:38:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:38:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:38:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:38:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_45 2021-07-10T09:38:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2021-07-10T09:38:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_45 128M 2021-07-10T09:38:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:38:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:38:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:38:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:38:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2021-07-10T09:38:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:38:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2021-07-10T09:38:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:38:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2021-07-10T09:38:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:38:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:38:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_45 2021-07-10T09:38:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:38:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:38:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:38:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:38:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:38:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 27726848 bytes 2021-07-10T09:38:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 40071168 bytes 2021-07-10T09:38:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 81934336 bytes 2021-07-10T09:38:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 128104448 bytes 2021-07-10T09:38:11.721 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9114370161951802) > weight (0.25) 2021-07-10T09:38:11.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4069336734653015) > weight (0.25) 2021-07-10T09:38:11.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9498251856917547) > weight (0.25) 2021-07-10T09:38:11.722 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2021-07-10T09:38:11.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2021-07-10T09:38:11.723 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2021-07-10T09:38:11.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi105.stderr:2021-07-10T09:38:11.723+0000 7f7218925700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 47984) UID: 1000 2021-07-10T09:38:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 147996672 bytes 2021-07-10T09:38:11.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:38:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 151415808 bytes 2021-07-10T09:38:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 224608256 bytes 2021-07-10T09:38:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 256199680 bytes 2021-07-10T09:38:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 275993600 bytes 2021-07-10T09:38:12.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 288916480 bytes 2021-07-10T09:38:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 338539520 bytes 2021-07-10T09:38:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 449382400 bytes 2021-07-10T09:38:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 461403136 bytes 2021-07-10T09:38:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 526504960 bytes 2021-07-10T09:38:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 563063808 bytes 2021-07-10T09:38:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 624152576 bytes 2021-07-10T09:38:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 736848896 bytes 2021-07-10T09:38:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 759464960 bytes 2021-07-10T09:38:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 790351872 bytes 2021-07-10T09:38:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:38:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:38:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:38:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:38:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:38:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_46 2021-07-10T09:38:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2021-07-10T09:38:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:38:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:38:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_46 128M 2021-07-10T09:38:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:38:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:38:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:38:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:38:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2021-07-10T09:38:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:38:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2021-07-10T09:38:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:38:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2021-07-10T09:38:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:38:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:38:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_46 2021-07-10T09:38:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:38:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:38:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=11 2021-07-10T09:38:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:38:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:38:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 82658304 bytes 2021-07-10T09:38:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 120292352 bytes 2021-07-10T09:38:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 200139776 bytes 2021-07-10T09:38:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 215840768 bytes 2021-07-10T09:38:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 253701120 bytes 2021-07-10T09:38:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 297348096 bytes 2021-07-10T09:38:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 342436864 bytes 2021-07-10T09:38:18.724 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2021-07-10T09:38:18.724 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:38:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:38:18.725 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2021-07-10T09:38:18.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2021-07-10T09:38:18.725 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2021-07-10T09:38:18.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2021-07-10T09:38:18.729 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2021-07-10T09:38:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 349877248 bytes 2021-07-10T09:38:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 400781312 bytes 2021-07-10T09:38:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 454249472 bytes 2021-07-10T09:38:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 521073664 bytes 2021-07-10T09:38:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 600521728 bytes 2021-07-10T09:38:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 655428608 bytes 2021-07-10T09:38:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 731039744 bytes 2021-07-10T09:38:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 758620160 bytes 2021-07-10T09:38:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 799880192 bytes 2021-07-10T09:38:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 874441728 bytes 2021-07-10T09:38:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 923494400 bytes 2021-07-10T09:38:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 945665024 bytes 2021-07-10T09:38:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 982735872 bytes 2021-07-10T09:38:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 1088423936 bytes 2021-07-10T09:38:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 1125594112 bytes 2021-07-10T09:38:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 1179992064 bytes 2021-07-10T09:38:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1261797376 bytes 2021-07-10T09:38:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1346758656 bytes 2021-07-10T09:38:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1406668800 bytes 2021-07-10T09:38:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1442471936 bytes 2021-07-10T09:38:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1503169536 bytes 2021-07-10T09:38:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1563446272 bytes 2021-07-10T09:38:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1592666112 bytes 2021-07-10T09:38:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1628578816 bytes 2021-07-10T09:38:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1661359104 bytes 2021-07-10T09:38:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1684857856 bytes 2021-07-10T09:38:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1749734400 bytes 2021-07-10T09:38:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1750348800 bytes 2021-07-10T09:38:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1784754176 bytes 2021-07-10T09:38:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1838726144 bytes 2021-07-10T09:38:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1863543808 bytes 2021-07-10T09:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:38:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:38:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_47 2021-07-10T09:38:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2021-07-10T09:38:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:38:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:38:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_47 128M 2021-07-10T09:38:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:38:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:38:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:38:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:38:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2021-07-10T09:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2021-07-10T09:38:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:38:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2021-07-10T09:38:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:38:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:38:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_47 2021-07-10T09:38:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:38:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:38:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=14 2021-07-10T09:38:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:38:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:38:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 20336640 bytes 2021-07-10T09:38:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 75165696 bytes 2021-07-10T09:38:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 145108992 bytes 2021-07-10T09:38:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 191025152 bytes 2021-07-10T09:38:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 191649792 bytes 2021-07-10T09:38:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 192274432 bytes 2021-07-10T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 252891136 bytes 2021-07-10T09:38:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 327994368 bytes 2021-07-10T09:38:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 344903680 bytes 2021-07-10T09:38:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 391873536 bytes 2021-07-10T09:38:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 446268416 bytes 2021-07-10T09:38:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 454129664 bytes 2021-07-10T09:38:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 470390784 bytes 2021-07-10T09:38:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 490458112 bytes 2021-07-10T09:38:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 516155392 bytes 2021-07-10T09:38:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 573701120 bytes 2021-07-10T09:38:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 618778624 bytes 2021-07-10T09:38:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 653474816 bytes 2021-07-10T09:38:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 773927936 bytes 2021-07-10T09:38:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 796677120 bytes 2021-07-10T09:38:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 818981888 bytes 2021-07-10T09:38:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 827153408 bytes 2021-07-10T09:38:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 911523840 bytes 2021-07-10T09:38:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 958337024 bytes 2021-07-10T09:38:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 984566784 bytes 2021-07-10T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1029408768 bytes 2021-07-10T09:38:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1104143360 bytes 2021-07-10T09:38:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1186086912 bytes 2021-07-10T09:38:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1196720128 bytes 2021-07-10T09:38:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1213278208 bytes 2021-07-10T09:38:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1286640640 bytes 2021-07-10T09:38:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1342390272 bytes 2021-07-10T09:38:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1366397952 bytes 2021-07-10T09:38:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1388112896 bytes 2021-07-10T09:38:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1394772992 bytes 2021-07-10T09:38:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1435054080 bytes 2021-07-10T09:38:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1488705536 bytes 2021-07-10T09:38:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1504557056 bytes 2021-07-10T09:38:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1593188352 bytes 2021-07-10T09:38:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1634366464 bytes 2021-07-10T09:38:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1675284480 bytes 2021-07-10T09:38:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1724998656 bytes 2021-07-10T09:38:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1784099840 bytes 2021-07-10T09:38:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1866366976 bytes 2021-07-10T09:38:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4500 IOs, 1912157184 bytes 2021-07-10T09:38:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4600 IOs, 1988279296 bytes 2021-07-10T09:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4700 IOs, 2059718656 bytes 2021-07-10T09:38:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4800 IOs, 2077165568 bytes 2021-07-10T09:38:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4900 IOs, 2096702464 bytes 2021-07-10T09:38:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:5000 IOs, 2097324032 bytes 2021-07-10T09:38:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:38:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:38:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:38:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:38:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:38:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_48 2021-07-10T09:38:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2021-07-10T09:38:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:38:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:38:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_48 128M 2021-07-10T09:38:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:38:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:38:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:38:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:38:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2021-07-10T09:38:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:38:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2021-07-10T09:38:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:38:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2021-07-10T09:38:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:38:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:38:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_48 2021-07-10T09:38:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:38:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:38:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=11 2021-07-10T09:38:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:38:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:38:42.895 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:38:42.891+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:38:42.896 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:38:42.892+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:38:42.896 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:38:42.893+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:38:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 72171520 bytes 2021-07-10T09:38:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 72787968 bytes 2021-07-10T09:38:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 158788608 bytes 2021-07-10T09:38:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 210963456 bytes 2021-07-10T09:38:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 242237440 bytes 2021-07-10T09:38:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 285052928 bytes 2021-07-10T09:38:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 306143232 bytes 2021-07-10T09:38:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 357817344 bytes 2021-07-10T09:38:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 405996544 bytes 2021-07-10T09:38:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 448952320 bytes 2021-07-10T09:38:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 487671808 bytes 2021-07-10T09:38:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 488278016 bytes 2021-07-10T09:38:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 531053568 bytes 2021-07-10T09:38:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 585390080 bytes 2021-07-10T09:38:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 686977024 bytes 2021-07-10T09:38:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 730799104 bytes 2021-07-10T09:38:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 771239936 bytes 2021-07-10T09:38:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 794237952 bytes 2021-07-10T09:38:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 862704640 bytes 2021-07-10T09:38:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 900537344 bytes 2021-07-10T09:38:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 916072448 bytes 2021-07-10T09:38:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 928990208 bytes 2021-07-10T09:38:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 951306240 bytes 2021-07-10T09:38:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 986128384 bytes 2021-07-10T09:38:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1026606080 bytes 2021-07-10T09:38:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1067385856 bytes 2021-07-10T09:38:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1086283776 bytes 2021-07-10T09:38:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1095183360 bytes 2021-07-10T09:38:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1148102656 bytes 2021-07-10T09:38:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1179067392 bytes 2021-07-10T09:38:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1190067200 bytes 2021-07-10T09:38:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1272065024 bytes 2021-07-10T09:38:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1323259904 bytes 2021-07-10T09:38:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1375301632 bytes 2021-07-10T09:38:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1428099072 bytes 2021-07-10T09:38:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1479508992 bytes 2021-07-10T09:38:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1549784064 bytes 2021-07-10T09:38:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1607996416 bytes 2021-07-10T09:38:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:38:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:38:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:38:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:38:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:38:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_49 2021-07-10T09:38:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2021-07-10T09:38:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:38:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:38:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_49 128M 2021-07-10T09:38:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:38:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:38:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:38:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:38:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:38:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2021-07-10T09:38:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:38:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2021-07-10T09:38:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:38:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:38:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2021-07-10T09:38:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:38:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:38:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_49 2021-07-10T09:38:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:38:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:38:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:38:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:38:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:38:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:38:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:38:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 14006272 bytes 2021-07-10T09:38:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 43639808 bytes 2021-07-10T09:38:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 71944192 bytes 2021-07-10T09:38:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 72567808 bytes 2021-07-10T09:38:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 110750720 bytes 2021-07-10T09:38:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 188496896 bytes 2021-07-10T09:38:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 225573888 bytes 2021-07-10T09:38:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 267798528 bytes 2021-07-10T09:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 354125824 bytes 2021-07-10T09:38:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 393373696 bytes 2021-07-10T09:38:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 423414784 bytes 2021-07-10T09:38:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 431228928 bytes 2021-07-10T09:38:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 450072576 bytes 2021-07-10T09:38:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 521243648 bytes 2021-07-10T09:38:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 578039808 bytes 2021-07-10T09:38:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 631360512 bytes 2021-07-10T09:38:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 653874176 bytes 2021-07-10T09:38:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 654497792 bytes 2021-07-10T09:38:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 688675840 bytes 2021-07-10T09:39:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 701533184 bytes 2021-07-10T09:39:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 710688768 bytes 2021-07-10T09:39:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 732836864 bytes 2021-07-10T09:39:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 775897088 bytes 2021-07-10T09:39:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 811275264 bytes 2021-07-10T09:39:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 846695424 bytes 2021-07-10T09:39:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 904146944 bytes 2021-07-10T09:39:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 904761344 bytes 2021-07-10T09:39:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 915914752 bytes 2021-07-10T09:39:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 916539392 bytes 2021-07-10T09:39:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:39:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:39:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:39:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_50 2021-07-10T09:39:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2021-07-10T09:39:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:39:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:39:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n snapshot ']' 2021-07-10T09:39:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_50 128M 2021-07-10T09:39:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:39:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:39:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 128M ']' 2021-07-10T09:39:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=128M 2021-07-10T09:39:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:39:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2021-07-10T09:39:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' image = image ']' 2021-07-10T09:39:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2021-07-10T09:39:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:39:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:39:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2021-07-10T09:39:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:39:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n '' ']' 2021-07-10T09:39:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_50 2021-07-10T09:39:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:39:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=14 2021-07-10T09:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 24697856 bytes 2021-07-10T09:39:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 65174528 bytes 2021-07-10T09:39:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 101882880 bytes 2021-07-10T09:39:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 147916800 bytes 2021-07-10T09:39:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 179955712 bytes 2021-07-10T09:39:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 202745856 bytes 2021-07-10T09:39:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 275211264 bytes 2021-07-10T09:39:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 351023104 bytes 2021-07-10T09:39:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 374448128 bytes 2021-07-10T09:39:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 451635200 bytes 2021-07-10T09:39:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 525828096 bytes 2021-07-10T09:39:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 558317568 bytes 2021-07-10T09:39:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 625568768 bytes 2021-07-10T09:39:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 640791552 bytes 2021-07-10T09:39:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 685774848 bytes 2021-07-10T09:39:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 717468672 bytes 2021-07-10T09:39:08.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 776567808 bytes 2021-07-10T09:39:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 811247616 bytes 2021-07-10T09:39:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 829000704 bytes 2021-07-10T09:39:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 855502848 bytes 2021-07-10T09:39:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 902003712 bytes 2021-07-10T09:39:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 915178496 bytes 2021-07-10T09:39:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 921817088 bytes 2021-07-10T09:39:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 1005453312 bytes 2021-07-10T09:39:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1054208000 bytes 2021-07-10T09:39:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1075907584 bytes 2021-07-10T09:39:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1110310912 bytes 2021-07-10T09:39:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1154289664 bytes 2021-07-10T09:39:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1216413696 bytes 2021-07-10T09:39:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1283438592 bytes 2021-07-10T09:39:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1386890240 bytes 2021-07-10T09:39:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1425242112 bytes 2021-07-10T09:39:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1456116736 bytes 2021-07-10T09:39:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1485219840 bytes 2021-07-10T09:39:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1558169600 bytes 2021-07-10T09:39:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1639708672 bytes 2021-07-10T09:39:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1719477248 bytes 2021-07-10T09:39:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1740503040 bytes 2021-07-10T09:39:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1741094912 bytes 2021-07-10T09:39:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1822412800 bytes 2021-07-10T09:39:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1895581696 bytes 2021-07-10T09:39:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1982551040 bytes 2021-07-10T09:39:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 2040218624 bytes 2021-07-10T09:39:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:39:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:39:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:39:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_pool_images cluster2 mirror 50 2021-07-10T09:39:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_count=50 2021-07-10T09:39:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local count 2021-07-10T09:39:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_count=0 2021-07-10T09:39:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 40 2021-07-10T09:39:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:39:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 1 -ne 1 2021-07-10T09:39:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2021-07-10T09:39:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 2 2021-07-10T09:39:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'images: ' 2021-07-10T09:39:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:39:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ count=50 2021-07-10T09:39:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 50 = 50 2021-07-10T09:39:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_pool_healthy cluster2 mirror 2021-07-10T09:39:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state 2021-07-10T09:39:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 40 2021-07-10T09:39:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:39:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 1 -ne 1 2021-07-10T09:39:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2021-07-10T09:39:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'image health:' 2021-07-10T09:39:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 3 2021-07-10T09:39:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:39:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=OK 2021-07-10T09:39:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test OK = ERROR 2021-07-10T09:39:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test OK = OK 2021-07-10T09:39:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_pool_images cluster1 mirror 50 2021-07-10T09:39:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_count=50 2021-07-10T09:39:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local count 2021-07-10T09:39:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_count=0 2021-07-10T09:39:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 40 2021-07-10T09:39:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:39:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 1 -ne 1 2021-07-10T09:39:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2021-07-10T09:39:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'images: ' 2021-07-10T09:39:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 2 2021-07-10T09:39:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:39:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ count=50 2021-07-10T09:39:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 50 = 50 2021-07-10T09:39:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_pool_healthy cluster1 mirror 2021-07-10T09:39:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state 2021-07-10T09:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 40 2021-07-10T09:39:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:39:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 1 -ne 1 2021-07-10T09:39:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2021-07-10T09:39:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'image health:' 2021-07-10T09:39:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 3 2021-07-10T09:39:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:39:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=OK 2021-07-10T09:39:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test OK = ERROR 2021-07-10T09:39:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test OK = OK 2021-07-10T09:39:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ testlog 'TEST: compare many images' 2021-07-10T09:39:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee -a /tmp/tmp.VSH8FeaLBn/rbd-mirror.test.log 2021-07-10T09:39:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ date '+%F %T' 2021-07-10T09:39:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo 2021-07-10 09:39:17 TEST: compare many images 2021-07-10T09:39:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10 09:39:17 TEST: compare many images 2021-07-10T09:39:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 50 2021-07-10T09:39:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_1 2021-07-10T09:39:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_1 snap 2021-07-10T09:39:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2021-07-10T09:39:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2021-07-10T09:39:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2021-07-10T09:39:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2021-07-10T09:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:39:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:39:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:39:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:39:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:39:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:39:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2021-07-10T09:39:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:39:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:39:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:39:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:39:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:39:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:39:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:39:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:39:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:39:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:39:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:39:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:39:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2021-07-10T09:39:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 3 closest matches: 2021-07-10T09:39:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_11 2021-07-10T09:39:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_15 2021-07-10T09:39:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_19 2021-07-10T09:39:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2021-07-10T09:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:39:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:39:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:39:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:39:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:39:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:39:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:39:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:39:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:39:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:39:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:39:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:39:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2021-07-10T09:39:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:39:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:39:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:39:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:39:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:39:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:39:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:39:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2021-07-10T09:39:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:39:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:39:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2021-07-10T09:39:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_1.status 2021-07-10T09:39:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_1.status 2021-07-10T09:39:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.status 2021-07-10T09:39:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_1.status 2021-07-10T09:39:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_1.status 2021-07-10T09:39:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.status 2021-07-10T09:39:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2021-07-10T09:39:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2021-07-10T09:39:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 119 2021-07-10T09:39:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.status 2021-07-10T09:39:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.status 2021-07-10T09:39:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2021-07-10T09:39:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:39:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=119 2021-07-10T09:39:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:39:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.status 2021-07-10T09:39:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.status 2021-07-10T09:39:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2021-07-10T09:39:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 119 = '' 2021-07-10T09:39:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:39:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.status 2021-07-10T09:39:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.status 2021-07-10T09:39:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2021-07-10T09:39:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 119 = '' 2021-07-10T09:39:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:39:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.status 2021-07-10T09:39:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.status 2021-07-10T09:39:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2021-07-10T09:39:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=119 2021-07-10T09:39:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 119 = 119 2021-07-10T09:39:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2021-07-10T09:39:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:39:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2021-07-10T09:39:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_1': 2021-07-10T09:39:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:39:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:39:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:39:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10b911f5135b 2021-07-10T09:39:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10b911f5135b 2021-07-10T09:39:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:39:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:39:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:39:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:39:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:31:05 2021 2021-07-10T09:39:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:31:05 2021 2021-07-10T09:39:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:31:05 2021 2021-07-10T09:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 26bc9239-9e96-42a6-95b5-071b3e0d944d 2021-07-10T09:39:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:39:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_1 snap 2021-07-10T09:39:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:39:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:39:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.export 2021-07-10T09:39:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.export 2021-07-10T09:39:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.export 2021-07-10T09:39:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.export 2021-07-10T09:39:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.export 2021-07-10T09:39:23.667 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:39:23.666+0000 7f4229178700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:39:23.668 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:39:23.667+0000 7f4226973700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:39:23.739 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:23.738+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:39:23.740 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:23.739+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:39:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.export 2021-07-10T09:39:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_1.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_1.export 2021-07-10T09:39:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_2 2021-07-10T09:39:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_2 snap 2021-07-10T09:39:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2021-07-10T09:39:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2021-07-10T09:39:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:39:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2021-07-10T09:39:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:39:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:39:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:39:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2021-07-10T09:39:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:39:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:39:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:39:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:39:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:39:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:39:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:39:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:39:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2021-07-10T09:39:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:39:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:39:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:39:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:39:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:39:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:39:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:39:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:39:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:39:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:39:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:39:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:39:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2021-07-10T09:39:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 5 closest matches: 2021-07-10T09:39:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_23 2021-07-10T09:39:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_25 2021-07-10T09:39:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_26 2021-07-10T09:39:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_27 2021-07-10T09:39:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_28 2021-07-10T09:39:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2021-07-10T09:39:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:39:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:39:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:39:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:39:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:39:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:39:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:39:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:39:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:39:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:39:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:39:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:39:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2021-07-10T09:39:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 2 closest matches: 2021-07-10T09:39:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_21 2021-07-10T09:39:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_29 2021-07-10T09:39:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2021-07-10T09:39:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:39:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:39:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:39:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:39:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:39:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:39:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:39:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:39:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:39:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:39:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:39:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:39:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2021-07-10T09:39:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 1 closest matches: 2021-07-10T09:39:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_22 2021-07-10T09:39:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2021-07-10T09:39:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:39:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:39:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:39:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:39:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:39:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:39:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:39:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:39:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:39:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:26.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:39:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:39:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:39:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2021-07-10T09:39:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:39:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:39:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:39:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:39:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:39:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:39:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:39:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2021-07-10T09:39:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:39:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:39:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2021-07-10T09:39:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_2.status 2021-07-10T09:39:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_2.status 2021-07-10T09:39:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.status 2021-07-10T09:39:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_2.status 2021-07-10T09:39:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_2.status 2021-07-10T09:39:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.status 2021-07-10T09:39:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2021-07-10T09:39:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2021-07-10T09:39:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 122 2021-07-10T09:39:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.status 2021-07-10T09:39:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.status 2021-07-10T09:39:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2021-07-10T09:39:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:39:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=122 2021-07-10T09:39:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:39:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.status 2021-07-10T09:39:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.status 2021-07-10T09:39:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2021-07-10T09:39:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 122 = '' 2021-07-10T09:39:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:39:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.status 2021-07-10T09:39:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.status 2021-07-10T09:39:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2021-07-10T09:39:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 122 = '' 2021-07-10T09:39:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:39:28.145 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:39:28.144+0000 7f4225971700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:39:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.status 2021-07-10T09:39:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.status 2021-07-10T09:39:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2021-07-10T09:39:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=122 2021-07-10T09:39:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 122 = 122 2021-07-10T09:39:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2021-07-10T09:39:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:39:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2021-07-10T09:39:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_2': 2021-07-10T09:39:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:39:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:39:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:39:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a480251e90 2021-07-10T09:39:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a480251e90 2021-07-10T09:39:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:39:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:39:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:39:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:39:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:31:18 2021 2021-07-10T09:39:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:31:18 2021 2021-07-10T09:39:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:31:18 2021 2021-07-10T09:39:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:39:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:39:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:39:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 19db4bfe-ff34-4b81-b85b-a2200ed19e14 2021-07-10T09:39:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:39:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_2 snap 2021-07-10T09:39:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:39:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:39:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.export 2021-07-10T09:39:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.export 2021-07-10T09:39:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.export 2021-07-10T09:39:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.export 2021-07-10T09:39:29.693 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:39:29.691+0000 7f69e7730700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:39:29.693 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2021-07-10T09:39:29.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2021-07-10T09:39:29.694 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2021-07-10T09:39:29.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:39:29.693+0000 7f7a21e13700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 49766) UID: 1000 2021-07-10T09:39:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.export 2021-07-10T09:39:29.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:39:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.export 2021-07-10T09:39:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_2.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_2.export 2021-07-10T09:39:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_3 2021-07-10T09:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_3 snap 2021-07-10T09:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2021-07-10T09:39:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2021-07-10T09:39:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:39:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:30.695 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2021-07-10T09:39:30.695 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:39:30.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:39:30.696 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2021-07-10T09:39:30.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:39:30.697 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:39:30.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:39:30.700 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2021-07-10T09:39:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2021-07-10T09:39:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:39:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:39:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:39:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2021-07-10T09:39:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:39:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:39:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:39:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:39:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:39:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:39:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:39:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:39:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2021-07-10T09:39:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:39:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:39:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:39:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:39:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:39:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:39:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:39:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:39:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:39:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:39:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2021-07-10T09:39:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 2 closest matches: 2021-07-10T09:39:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_32 2021-07-10T09:39:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_36 2021-07-10T09:39:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2021-07-10T09:39:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:39:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:39:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:39:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:39:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:39:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:39:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:39:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:39:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:39:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:39:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:39:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:39:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2021-07-10T09:39:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 2 closest matches: 2021-07-10T09:39:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_33 2021-07-10T09:39:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_37 2021-07-10T09:39:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2021-07-10T09:39:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:39:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:39:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:39:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:39:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:39:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:39:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:39:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:39:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:39:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:39:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:39:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:39:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2021-07-10T09:39:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 5 closest matches: 2021-07-10T09:39:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_30 2021-07-10T09:39:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_31 2021-07-10T09:39:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_34 2021-07-10T09:39:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_35 2021-07-10T09:39:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_38 2021-07-10T09:39:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2021-07-10T09:39:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:39:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:39:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:39:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:39:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:39:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:39:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:39:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:39:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:39:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:39:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:39:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:39:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2021-07-10T09:39:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:39:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:39:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:39:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:39:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2021-07-10T09:39:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:39:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:39:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2021-07-10T09:39:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_3.status 2021-07-10T09:39:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_3.status 2021-07-10T09:39:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.status 2021-07-10T09:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_3.status 2021-07-10T09:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_3.status 2021-07-10T09:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2021-07-10T09:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2021-07-10T09:39:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 125 2021-07-10T09:39:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.status 2021-07-10T09:39:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.status 2021-07-10T09:39:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2021-07-10T09:39:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:39:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=125 2021-07-10T09:39:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:39:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2021-07-10T09:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 125 = '' 2021-07-10T09:39:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:39:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2021-07-10T09:39:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 125 = '' 2021-07-10T09:39:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:39:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2021-07-10T09:39:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 125 = '' 2021-07-10T09:39:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:39:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.status 2021-07-10T09:39:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2021-07-10T09:39:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=125 2021-07-10T09:39:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 125 = 125 2021-07-10T09:39:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2021-07-10T09:39:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:39:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2021-07-10T09:39:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_3': 2021-07-10T09:39:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:39:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:39:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:39:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f2e820799 2021-07-10T09:39:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f2e820799 2021-07-10T09:39:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:39:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:39:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:39:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:39:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:31:24 2021 2021-07-10T09:39:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:31:24 2021 2021-07-10T09:39:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:31:24 2021 2021-07-10T09:39:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:39:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:39:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:39:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 336108ed-7ac8-4c4b-a75a-cdfb90bf3e5e 2021-07-10T09:39:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:39:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_3 snap 2021-07-10T09:39:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:39:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:39:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.export 2021-07-10T09:39:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.export 2021-07-10T09:39:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.export 2021-07-10T09:39:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.export 2021-07-10T09:39:37.031 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:37.030+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:39:37.033 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:37.030+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:39:37.033 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:37.030+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:39:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.export 2021-07-10T09:39:37.254 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:37.251+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:39:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.export 2021-07-10T09:39:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_3.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_3.export 2021-07-10T09:39:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_4 2021-07-10T09:39:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_4 snap 2021-07-10T09:39:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2021-07-10T09:39:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2021-07-10T09:39:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:39:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2021-07-10T09:39:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:39:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:39:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:39:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2021-07-10T09:39:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:39:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:39:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:39:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:39:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:39:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:39:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:39:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:39:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2021-07-10T09:39:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:39:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:39:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:39:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:39:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:39:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:39:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:39:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:39:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:39:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:39:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:39:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:39:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2021-07-10T09:39:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 2 closest matches: 2021-07-10T09:39:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_40 2021-07-10T09:39:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_44 2021-07-10T09:39:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2021-07-10T09:39:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:39:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:39:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:39:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:39:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2021-07-10T09:39:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 4 closest matches: 2021-07-10T09:39:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_41 2021-07-10T09:39:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_43 2021-07-10T09:39:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_45 2021-07-10T09:39:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_49 2021-07-10T09:39:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2021-07-10T09:39:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:39:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:39:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:39:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:39:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:39:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2021-07-10T09:39:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:39:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:39:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:39:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:39:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:39:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:39:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:39:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2021-07-10T09:39:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2021-07-10T09:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_4.status 2021-07-10T09:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_4.status 2021-07-10T09:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.status 2021-07-10T09:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_4.status 2021-07-10T09:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_4.status 2021-07-10T09:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2021-07-10T09:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2021-07-10T09:39:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 128 2021-07-10T09:39:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.status 2021-07-10T09:39:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.status 2021-07-10T09:39:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2021-07-10T09:39:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:39:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=128 2021-07-10T09:39:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:39:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2021-07-10T09:39:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=71 2021-07-10T09:39:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 128 = 71 2021-07-10T09:39:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:39:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2021-07-10T09:39:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=71 2021-07-10T09:39:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 128 = 71 2021-07-10T09:39:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:39:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2021-07-10T09:39:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 128 = '' 2021-07-10T09:39:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:39:42.943 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:39:42.941+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:39:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.status 2021-07-10T09:39:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2021-07-10T09:39:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=128 2021-07-10T09:39:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 128 = 128 2021-07-10T09:39:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2021-07-10T09:39:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:39:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2021-07-10T09:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_4': 2021-07-10T09:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:39:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:39:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:39:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12a5aec10eb 2021-07-10T09:39:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12a5aec10eb 2021-07-10T09:39:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:39:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:31:35 2021 2021-07-10T09:39:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:31:35 2021 2021-07-10T09:39:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:31:35 2021 2021-07-10T09:39:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:39:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:39:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:39:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 9afd022c-553e-4abb-9f19-970ff5549ba8 2021-07-10T09:39:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:39:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_4 snap 2021-07-10T09:39:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:39:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:39:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.export 2021-07-10T09:39:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.export 2021-07-10T09:39:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.export 2021-07-10T09:39:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.export 2021-07-10T09:39:45.315 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:45.313+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:39:45.316 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:45.313+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:39:45.316 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:45.314+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:39:45.317 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:45.314+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:39:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.export 2021-07-10T09:39:45.414 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:45.413+0000 7f317385f700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:39:45.419 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:45.414+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:39:45.510 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:45.508+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:39:45.511 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:45.508+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:39:45.511 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:45.508+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:39:45.511 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:45.508+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:39:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.export 2021-07-10T09:39:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_4.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_4.export 2021-07-10T09:39:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_5 2021-07-10T09:39:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_5 snap 2021-07-10T09:39:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2021-07-10T09:39:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2021-07-10T09:39:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:39:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2021-07-10T09:39:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:39:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:39:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:39:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2021-07-10T09:39:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:39:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:39:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:39:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:39:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:39:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:39:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:39:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:39:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2021-07-10T09:39:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:39:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:39:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:39:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:39:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:39:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:39:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:39:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:39:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:39:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:39:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:39:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:39:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2021-07-10T09:39:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2021-07-10T09:39:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:39:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2021-07-10T09:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:39:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:39:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:39:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:39:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:39:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:39:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:39:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:39:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2021-07-10T09:39:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2021-07-10T09:39:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:39:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2021-07-10T09:39:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:39:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:39:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:39:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:39:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2021-07-10T09:39:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 1 closest matches: 2021-07-10T09:39:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_50 2021-07-10T09:39:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2021-07-10T09:39:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:39:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:39:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:39:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:39:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:39:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:39:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:39:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:39:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:39:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:39:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:39:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:39:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2021-07-10T09:39:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:39:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:39:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:39:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:39:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:39:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:39:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:39:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2021-07-10T09:39:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:39:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:39:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2021-07-10T09:39:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_5.status 2021-07-10T09:39:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_5.status 2021-07-10T09:39:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.status 2021-07-10T09:39:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_5.status 2021-07-10T09:39:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_5.status 2021-07-10T09:39:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.status 2021-07-10T09:39:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2021-07-10T09:39:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2021-07-10T09:39:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 131 2021-07-10T09:39:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.status 2021-07-10T09:39:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.status 2021-07-10T09:39:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2021-07-10T09:39:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:39:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=131 2021-07-10T09:39:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:39:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.status 2021-07-10T09:39:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.status 2021-07-10T09:39:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2021-07-10T09:39:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:48.962 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:39:48.960+0000 7f69f674e700 -1 reset not still connected to 0x563078e02680 2021-07-10T09:39:48.962 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:39:48.960+0000 7f69f674e700 -1 reset not still connected to 0x563079820270 2021-07-10T09:39:48.963 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:39:48.960+0000 7f69f674e700 -1 reset not still connected to 0x5630cbbc4750 2021-07-10T09:39:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 131 = '' 2021-07-10T09:39:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:39:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.status 2021-07-10T09:39:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.status 2021-07-10T09:39:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2021-07-10T09:39:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 131 = '' 2021-07-10T09:39:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.status 2021-07-10T09:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.status 2021-07-10T09:39:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2021-07-10T09:39:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=131 2021-07-10T09:39:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 131 = 131 2021-07-10T09:39:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2021-07-10T09:39:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:39:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2021-07-10T09:39:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_5': 2021-07-10T09:39:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:39:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:39:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:39:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 10b9a73a3552 2021-07-10T09:39:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.10b9a73a3552 2021-07-10T09:39:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:39:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:39:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:39:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:39:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:31:49 2021 2021-07-10T09:39:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:31:49 2021 2021-07-10T09:39:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:31:49 2021 2021-07-10T09:39:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:39:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:39:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:39:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 83b7f2a3-dd56-45d9-ab45-9bec5a241132 2021-07-10T09:39:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:39:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_5 snap 2021-07-10T09:39:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:39:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:39:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.export 2021-07-10T09:39:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.export 2021-07-10T09:39:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.export 2021-07-10T09:39:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.export 2021-07-10T09:39:51.361 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:51.359+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:39:51.361 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:51.359+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:39:51.425 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:51.422+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:39:51.425 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:39:51.422+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:39:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.export 2021-07-10T09:39:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.export 2021-07-10T09:39:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_5.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_5.export 2021-07-10T09:39:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_6 2021-07-10T09:39:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_6 snap 2021-07-10T09:39:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2021-07-10T09:39:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2021-07-10T09:39:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:39:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2021-07-10T09:39:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:39:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:39:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:39:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2021-07-10T09:39:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:39:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:39:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:39:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:39:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:39:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:39:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:39:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:39:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2021-07-10T09:39:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:39:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:39:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:39:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:39:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:39:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:39:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:39:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:39:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:39:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:39:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:39:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:39:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2021-07-10T09:39:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2021-07-10T09:39:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2021-07-10T09:39:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:39:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:39:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:39:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:39:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:39:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:39:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:39:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:39:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:39:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:39:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:39:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:39:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2021-07-10T09:39:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2021-07-10T09:39:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:39:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2021-07-10T09:39:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:39:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:39:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:39:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:39:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:39:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:39:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:39:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:39:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:39:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:39:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:39:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:39:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2021-07-10T09:39:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2021-07-10T09:39:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:39:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_6 2021-07-10T09:39:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:39:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:39:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:39:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:39:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:39:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:39:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:39:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:39:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:39:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:39:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:39:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:39:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:39:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:39:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:39:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_6 2021-07-10T09:39:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:39:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:39:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:39:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:39:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:39:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:39:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2021-07-10T09:39:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:39:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:39:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2021-07-10T09:39:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:39:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_6.status 2021-07-10T09:39:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_6.status 2021-07-10T09:39:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.status 2021-07-10T09:39:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_6.status 2021-07-10T09:39:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_6.status 2021-07-10T09:39:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:39:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.status 2021-07-10T09:39:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2021-07-10T09:39:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2021-07-10T09:39:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 134 2021-07-10T09:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.status 2021-07-10T09:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.status 2021-07-10T09:39:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2021-07-10T09:39:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:39:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=134 2021-07-10T09:39:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:39:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.status 2021-07-10T09:39:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.status 2021-07-10T09:39:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2021-07-10T09:39:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 134 = '' 2021-07-10T09:39:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:39:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.status 2021-07-10T09:39:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.status 2021-07-10T09:39:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2021-07-10T09:39:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:39:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:39:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:39:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:39:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:39:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 134 = '' 2021-07-10T09:39:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:39:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:39:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.status 2021-07-10T09:39:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.status 2021-07-10T09:39:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2021-07-10T09:39:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:39:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:39:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=134 2021-07-10T09:39:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 134 = 134 2021-07-10T09:39:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2021-07-10T09:39:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:39:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2021-07-10T09:39:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_6': 2021-07-10T09:39:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:39:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:39:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:39:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a476d81ac6 2021-07-10T09:39:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a476d81ac6 2021-07-10T09:39:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:39:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:39:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:39:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:39:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:31:56 2021 2021-07-10T09:39:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:31:56 2021 2021-07-10T09:39:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:31:56 2021 2021-07-10T09:39:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:39:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:39:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:39:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 666f9785-2bd5-480e-b89f-63bfc1687e18 2021-07-10T09:39:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:39:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_6 snap 2021-07-10T09:39:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:39:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:39:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.export 2021-07-10T09:39:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.export 2021-07-10T09:39:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.export 2021-07-10T09:39:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.export 2021-07-10T09:39:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.export 2021-07-10T09:39:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:57.576+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:39:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:57.576+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:39:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:57.576+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:39:57.579 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:39:57.578+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:39:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:39:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.export 2021-07-10T09:39:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_6.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_6.export 2021-07-10T09:39:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:39:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:39:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_7 2021-07-10T09:39:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_7 snap 2021-07-10T09:39:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:39:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:39:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:39:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:39:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2021-07-10T09:39:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:39:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2021-07-10T09:39:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:39:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:39:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:39:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:39:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2021-07-10T09:39:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:39:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:39:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:39:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:39:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:39:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:39:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2021-07-10T09:39:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:39:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:39:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:39:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:39:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:39:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:39:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:39:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:39:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:39:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2021-07-10T09:39:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:39:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:39:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:39:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:39:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:39:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:39:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:39:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:39:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:39:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:39:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:39:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:39:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:39:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:39:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:39:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2021-07-10T09:39:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2021-07-10T09:39:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:39:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2021-07-10T09:39:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:39:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:39:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:39:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:39:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:39:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:39:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:39:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:39:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:39:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:39:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:39:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:39:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:39:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:39:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2021-07-10T09:39:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2021-07-10T09:39:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:39:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:39:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:39:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2021-07-10T09:39:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:39:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:39:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:39:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:39:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:39:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:39:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:39:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:39:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:39:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:39:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:39:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:39:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:39:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:39:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:39:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:39:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:39:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:39:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:39:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:39:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:39:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:39:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:39:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:39:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:39:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:39:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2021-07-10T09:40:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2021-07-10T09:40:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2021-07-10T09:40:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:40:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:40:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:40:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:40:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:40:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:40:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:40:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:40:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:40:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:40:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:40:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:40:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:40:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:40:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:40:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:40:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2021-07-10T09:40:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2021-07-10T09:40:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2021-07-10T09:40:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_7.status 2021-07-10T09:40:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_7.status 2021-07-10T09:40:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.status 2021-07-10T09:40:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_7.status 2021-07-10T09:40:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_7.status 2021-07-10T09:40:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2021-07-10T09:40:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2021-07-10T09:40:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 137 2021-07-10T09:40:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.status 2021-07-10T09:40:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.status 2021-07-10T09:40:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2021-07-10T09:40:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=137 2021-07-10T09:40:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2021-07-10T09:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 137 = '' 2021-07-10T09:40:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2021-07-10T09:40:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 137 = '' 2021-07-10T09:40:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:01.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.899301653086327) > weight (0.25) 2021-07-10T09:40:01.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7825619964669005) > weight (0.25) 2021-07-10T09:40:01.731 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2683986827683431) > weight (0.25) 2021-07-10T09:40:01.731 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9076743920803593) > weight (0.25) 2021-07-10T09:40:01.731 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2021-07-10T09:40:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2021-07-10T09:40:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 137 = '' 2021-07-10T09:40:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:40:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.status 2021-07-10T09:40:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2021-07-10T09:40:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=137 2021-07-10T09:40:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 137 = 137 2021-07-10T09:40:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2021-07-10T09:40:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2021-07-10T09:40:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_7': 2021-07-10T09:40:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d42f5b8382 2021-07-10T09:40:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d42f5b8382 2021-07-10T09:40:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:32:05 2021 2021-07-10T09:40:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:32:05 2021 2021-07-10T09:40:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:32:05 2021 2021-07-10T09:40:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 5dd9e49d-3d37-4c87-a320-3a5a40e93fd4 2021-07-10T09:40:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_7 snap 2021-07-10T09:40:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:40:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.export 2021-07-10T09:40:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.export 2021-07-10T09:40:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.export 2021-07-10T09:40:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.export 2021-07-10T09:40:05.151 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:05.150+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:40:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.export 2021-07-10T09:40:05.244 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:05.243+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:05.244 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:05.243+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.export 2021-07-10T09:40:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_7.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_7.export 2021-07-10T09:40:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_8 2021-07-10T09:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_8 snap 2021-07-10T09:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2021-07-10T09:40:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2021-07-10T09:40:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2021-07-10T09:40:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2021-07-10T09:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2021-07-10T09:40:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2021-07-10T09:40:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2021-07-10T09:40:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2021-07-10T09:40:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:40:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:40:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:40:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:40:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:40:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:40:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:40:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:40:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:40:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:40:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:40:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:40:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2021-07-10T09:40:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2021-07-10T09:40:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2021-07-10T09:40:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:40:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:40:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:40:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:40:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:40:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:40:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:40:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:40:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:40:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:40:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:40:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:40:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2021-07-10T09:40:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2021-07-10T09:40:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2021-07-10T09:40:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:40:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:40:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:40:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:40:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:40:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:40:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:40:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:40:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:40:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:40:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:40:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:40:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:40:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:40:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:40:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:40:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2021-07-10T09:40:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2021-07-10T09:40:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2021-07-10T09:40:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_8.status 2021-07-10T09:40:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_8.status 2021-07-10T09:40:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.status 2021-07-10T09:40:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_8.status 2021-07-10T09:40:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_8.status 2021-07-10T09:40:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.status 2021-07-10T09:40:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2021-07-10T09:40:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2021-07-10T09:40:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 140 2021-07-10T09:40:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.status 2021-07-10T09:40:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.status 2021-07-10T09:40:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2021-07-10T09:40:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=140 2021-07-10T09:40:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.status 2021-07-10T09:40:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.status 2021-07-10T09:40:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2021-07-10T09:40:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 140 = '' 2021-07-10T09:40:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.status 2021-07-10T09:40:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.status 2021-07-10T09:40:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2021-07-10T09:40:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 140 = '' 2021-07-10T09:40:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.status 2021-07-10T09:40:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.status 2021-07-10T09:40:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2021-07-10T09:40:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=140 2021-07-10T09:40:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 140 = 140 2021-07-10T09:40:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2021-07-10T09:40:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2021-07-10T09:40:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_8': 2021-07-10T09:40:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f2a2e7451 2021-07-10T09:40:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f2a2e7451 2021-07-10T09:40:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:32:12 2021 2021-07-10T09:40:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:32:12 2021 2021-07-10T09:40:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:32:12 2021 2021-07-10T09:40:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: d654db96-c195-45c5-bbd4-41b6852de973 2021-07-10T09:40:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_8 snap 2021-07-10T09:40:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:40:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.export 2021-07-10T09:40:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.export 2021-07-10T09:40:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.export 2021-07-10T09:40:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.export 2021-07-10T09:40:11.358 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:11.356+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:40:11.417 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:11.414+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:40:11.422 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:11.415+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:40:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.export 2021-07-10T09:40:11.701 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:11.700+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:40:11.702 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:11.700+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:40:11.702 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:11.701+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.export 2021-07-10T09:40:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_8.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_8.export 2021-07-10T09:40:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_9 2021-07-10T09:40:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_9 snap 2021-07-10T09:40:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2021-07-10T09:40:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2021-07-10T09:40:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2021-07-10T09:40:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2021-07-10T09:40:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2021-07-10T09:40:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2021-07-10T09:40:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2021-07-10T09:40:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2021-07-10T09:40:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:40:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:40:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:40:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:40:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:40:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:40:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:40:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:40:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:40:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:40:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2021-07-10T09:40:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2021-07-10T09:40:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2021-07-10T09:40:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_9.status 2021-07-10T09:40:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_9.status 2021-07-10T09:40:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.status 2021-07-10T09:40:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_9.status 2021-07-10T09:40:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_9.status 2021-07-10T09:40:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.status 2021-07-10T09:40:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2021-07-10T09:40:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2021-07-10T09:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 143 2021-07-10T09:40:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.status 2021-07-10T09:40:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.status 2021-07-10T09:40:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2021-07-10T09:40:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=143 2021-07-10T09:40:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.status 2021-07-10T09:40:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.status 2021-07-10T09:40:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2021-07-10T09:40:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 143 = '' 2021-07-10T09:40:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.status 2021-07-10T09:40:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.status 2021-07-10T09:40:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2021-07-10T09:40:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 143 = '' 2021-07-10T09:40:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.status 2021-07-10T09:40:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.status 2021-07-10T09:40:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2021-07-10T09:40:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=143 2021-07-10T09:40:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 143 = 143 2021-07-10T09:40:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2021-07-10T09:40:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2021-07-10T09:40:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_9': 2021-07-10T09:40:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12a5fb06174f 2021-07-10T09:40:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12a5fb06174f 2021-07-10T09:40:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:32:18 2021 2021-07-10T09:40:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:32:18 2021 2021-07-10T09:40:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:32:18 2021 2021-07-10T09:40:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 99dbec2f-3ec0-48e7-9063-1774ffe01963 2021-07-10T09:40:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_9 snap 2021-07-10T09:40:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:40:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.export 2021-07-10T09:40:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.export 2021-07-10T09:40:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.export 2021-07-10T09:40:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.export 2021-07-10T09:40:17.425 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:40:17.424+0000 7f5116298700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:40:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.export 2021-07-10T09:40:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:17.600+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:17.668 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:40:17.667+0000 7f422b17c700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2021-07-10T09:40:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.export 2021-07-10T09:40:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_9.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_9.export 2021-07-10T09:40:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_10 2021-07-10T09:40:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_10 snap 2021-07-10T09:40:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2021-07-10T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2021-07-10T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2021-07-10T09:40:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2021-07-10T09:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2021-07-10T09:40:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2021-07-10T09:40:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2021-07-10T09:40:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2021-07-10T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:40:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:40:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:40:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:40:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:40:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:40:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:40:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:40:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2021-07-10T09:40:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2021-07-10T09:40:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2021-07-10T09:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:40:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:40:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:40:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:40:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:40:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:40:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:40:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:40:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:40:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:40:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:40:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2021-07-10T09:40:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2021-07-10T09:40:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2021-07-10T09:40:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_10.status 2021-07-10T09:40:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_10.status 2021-07-10T09:40:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.status 2021-07-10T09:40:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_10.status 2021-07-10T09:40:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_10.status 2021-07-10T09:40:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.status 2021-07-10T09:40:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2021-07-10T09:40:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2021-07-10T09:40:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 146 2021-07-10T09:40:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.status 2021-07-10T09:40:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.status 2021-07-10T09:40:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2021-07-10T09:40:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=146 2021-07-10T09:40:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.status 2021-07-10T09:40:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.status 2021-07-10T09:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2021-07-10T09:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 146 = '' 2021-07-10T09:40:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.status 2021-07-10T09:40:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.status 2021-07-10T09:40:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2021-07-10T09:40:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 146 = '' 2021-07-10T09:40:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.status 2021-07-10T09:40:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.status 2021-07-10T09:40:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2021-07-10T09:40:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=146 2021-07-10T09:40:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 146 = 146 2021-07-10T09:40:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2021-07-10T09:40:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2021-07-10T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_10': 2021-07-10T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f470a9ec 2021-07-10T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f470a9ec 2021-07-10T09:40:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:32:27 2021 2021-07-10T09:40:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:32:27 2021 2021-07-10T09:40:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:32:27 2021 2021-07-10T09:40:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: ec54d8e1-58c7-4033-a53d-9b1c54d31262 2021-07-10T09:40:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_10 snap 2021-07-10T09:40:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:40:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.export 2021-07-10T09:40:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.export 2021-07-10T09:40:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.export 2021-07-10T09:40:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.export 2021-07-10T09:40:23.425 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:23.424+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:40:23.426 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:23.425+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:40:23.427 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:23.425+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:40:23.428 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:23.426+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2021-07-10T09:40:23.480 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:23.479+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:40:23.481 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:23.480+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:40:23.482 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:23.481+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:40:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.export 2021-07-10T09:40:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.export 2021-07-10T09:40:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_10.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_10.export 2021-07-10T09:40:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_11 2021-07-10T09:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_11 snap 2021-07-10T09:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2021-07-10T09:40:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2021-07-10T09:40:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2021-07-10T09:40:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2021-07-10T09:40:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2021-07-10T09:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2021-07-10T09:40:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2021-07-10T09:40:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2021-07-10T09:40:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_11.status 2021-07-10T09:40:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_11.status 2021-07-10T09:40:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.status 2021-07-10T09:40:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_11.status 2021-07-10T09:40:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_11.status 2021-07-10T09:40:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.status 2021-07-10T09:40:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2021-07-10T09:40:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2021-07-10T09:40:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 149 2021-07-10T09:40:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.status 2021-07-10T09:40:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.status 2021-07-10T09:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2021-07-10T09:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=149 2021-07-10T09:40:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.status 2021-07-10T09:40:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.status 2021-07-10T09:40:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2021-07-10T09:40:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 149 = '' 2021-07-10T09:40:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.status 2021-07-10T09:40:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.status 2021-07-10T09:40:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2021-07-10T09:40:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 149 = '' 2021-07-10T09:40:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.status 2021-07-10T09:40:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.status 2021-07-10T09:40:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2021-07-10T09:40:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=149 2021-07-10T09:40:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 149 = 149 2021-07-10T09:40:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2021-07-10T09:40:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2021-07-10T09:40:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_11': 2021-07-10T09:40:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12a5f6f51b0b 2021-07-10T09:40:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12a5f6f51b0b 2021-07-10T09:40:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:32:34 2021 2021-07-10T09:40:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:32:34 2021 2021-07-10T09:40:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:32:34 2021 2021-07-10T09:40:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 2e6dff40-cd67-4488-8fba-c9b06b23089b 2021-07-10T09:40:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_11 snap 2021-07-10T09:40:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:40:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.export 2021-07-10T09:40:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.export 2021-07-10T09:40:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.export 2021-07-10T09:40:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.export 2021-07-10T09:40:29.426 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:29.424+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:40:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.export 2021-07-10T09:40:29.669 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:29.669+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:29.670 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:29.670+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:40:29.673 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:29.673+0000 7f317986b700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:29.741 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:29.740+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:40:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.export 2021-07-10T09:40:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_11.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_11.export 2021-07-10T09:40:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_12 2021-07-10T09:40:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_12 snap 2021-07-10T09:40:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2021-07-10T09:40:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2021-07-10T09:40:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2021-07-10T09:40:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2021-07-10T09:40:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2021-07-10T09:40:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2021-07-10T09:40:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2021-07-10T09:40:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2021-07-10T09:40:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:40:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:40:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:40:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:40:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:40:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:40:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:40:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:40:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:40:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:40:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:40:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:40:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2021-07-10T09:40:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2021-07-10T09:40:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2021-07-10T09:40:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_12.status 2021-07-10T09:40:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_12.status 2021-07-10T09:40:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.status 2021-07-10T09:40:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_12.status 2021-07-10T09:40:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_12.status 2021-07-10T09:40:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.status 2021-07-10T09:40:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2021-07-10T09:40:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2021-07-10T09:40:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 152 2021-07-10T09:40:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.status 2021-07-10T09:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.status 2021-07-10T09:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2021-07-10T09:40:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=152 2021-07-10T09:40:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.status 2021-07-10T09:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.status 2021-07-10T09:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2021-07-10T09:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 152 = '' 2021-07-10T09:40:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.status 2021-07-10T09:40:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.status 2021-07-10T09:40:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2021-07-10T09:40:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 152 = '' 2021-07-10T09:40:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.status 2021-07-10T09:40:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.status 2021-07-10T09:40:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2021-07-10T09:40:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=152 2021-07-10T09:40:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 152 = 152 2021-07-10T09:40:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2021-07-10T09:40:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2021-07-10T09:40:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_12': 2021-07-10T09:40:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d4821faf69 2021-07-10T09:40:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d4821faf69 2021-07-10T09:40:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:32:40 2021 2021-07-10T09:40:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:32:40 2021 2021-07-10T09:40:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:32:40 2021 2021-07-10T09:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: dda766d7-e6db-42cc-a154-1c727bfc33a6 2021-07-10T09:40:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_12 snap 2021-07-10T09:40:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.export 2021-07-10T09:40:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.export 2021-07-10T09:40:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.export 2021-07-10T09:40:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.export 2021-07-10T09:40:35.354 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:40:35.352+0000 7f69e7f31700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:40:35.463 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:35.461+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:40:35.464 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:35.462+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:40:35.464 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:35.462+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:40:35.465 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:35.464+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:40:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.export 2021-07-10T09:40:35.560 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:40:35.559+0000 7f72cc89c700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:40:35.560 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:40:35.559+0000 7f72c608f700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:40:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.export 2021-07-10T09:40:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_12.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_12.export 2021-07-10T09:40:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_13 2021-07-10T09:40:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_13 snap 2021-07-10T09:40:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2021-07-10T09:40:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2021-07-10T09:40:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2021-07-10T09:40:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2021-07-10T09:40:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2021-07-10T09:40:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2021-07-10T09:40:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2021-07-10T09:40:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2021-07-10T09:40:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:40:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:40:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:40:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:40:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:40:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:40:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:40:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:40:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:40:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:40:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:40:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:40:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2021-07-10T09:40:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2021-07-10T09:40:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2021-07-10T09:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_13.status 2021-07-10T09:40:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_13.status 2021-07-10T09:40:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.status 2021-07-10T09:40:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_13.status 2021-07-10T09:40:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_13.status 2021-07-10T09:40:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.status 2021-07-10T09:40:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2021-07-10T09:40:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2021-07-10T09:40:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 155 2021-07-10T09:40:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.status 2021-07-10T09:40:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.status 2021-07-10T09:40:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2021-07-10T09:40:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=155 2021-07-10T09:40:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.status 2021-07-10T09:40:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.status 2021-07-10T09:40:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2021-07-10T09:40:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 155 = '' 2021-07-10T09:40:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.status 2021-07-10T09:40:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.status 2021-07-10T09:40:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2021-07-10T09:40:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 155 = '' 2021-07-10T09:40:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.status 2021-07-10T09:40:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.status 2021-07-10T09:40:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2021-07-10T09:40:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=155 2021-07-10T09:40:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 155 = 155 2021-07-10T09:40:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2021-07-10T09:40:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2021-07-10T09:40:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_13': 2021-07-10T09:40:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a48002f06b 2021-07-10T09:40:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a48002f06b 2021-07-10T09:40:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:32:54 2021 2021-07-10T09:40:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:32:54 2021 2021-07-10T09:40:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:32:54 2021 2021-07-10T09:40:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: cea3e863-265e-4c08-8ea9-92acf60cff4f 2021-07-10T09:40:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_13 snap 2021-07-10T09:40:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:40:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.export 2021-07-10T09:40:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.export 2021-07-10T09:40:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.export 2021-07-10T09:40:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.export 2021-07-10T09:40:41.383 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:40:41.382+0000 7f5116298700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:40:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.export 2021-07-10T09:40:41.597 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:40:41.596+0000 7f422416e700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:40:41.680 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:41.676+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:40:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.export 2021-07-10T09:40:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_13.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_13.export 2021-07-10T09:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_14 2021-07-10T09:40:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_14 snap 2021-07-10T09:40:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2021-07-10T09:40:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2021-07-10T09:40:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2021-07-10T09:40:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2021-07-10T09:40:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2021-07-10T09:40:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2021-07-10T09:40:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2021-07-10T09:40:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2021-07-10T09:40:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:40:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:40:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:40:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:40:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:40:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:40:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:40:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:40:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:40:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:40:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:40:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:40:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2021-07-10T09:40:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2021-07-10T09:40:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2021-07-10T09:40:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:40:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:40:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:40:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:40:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:40:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:40:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:40:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:40:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:40:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:40:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:40:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2021-07-10T09:40:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2021-07-10T09:40:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2021-07-10T09:40:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:40:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:40:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:40:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:40:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:40:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:40:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:40:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:40:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:40:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:40:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:40:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:40:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:40:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:40:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2021-07-10T09:40:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2021-07-10T09:40:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2021-07-10T09:40:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_14.status 2021-07-10T09:40:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_14.status 2021-07-10T09:40:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.status 2021-07-10T09:40:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_14.status 2021-07-10T09:40:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_14.status 2021-07-10T09:40:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.status 2021-07-10T09:40:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2021-07-10T09:40:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2021-07-10T09:40:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 158 2021-07-10T09:40:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.status 2021-07-10T09:40:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.status 2021-07-10T09:40:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2021-07-10T09:40:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=158 2021-07-10T09:40:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.status 2021-07-10T09:40:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.status 2021-07-10T09:40:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2021-07-10T09:40:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 158 = '' 2021-07-10T09:40:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.status 2021-07-10T09:40:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.status 2021-07-10T09:40:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2021-07-10T09:40:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 158 = '' 2021-07-10T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.status 2021-07-10T09:40:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.status 2021-07-10T09:40:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2021-07-10T09:40:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=158 2021-07-10T09:40:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 158 = 158 2021-07-10T09:40:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2021-07-10T09:40:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2021-07-10T09:40:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_14': 2021-07-10T09:40:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122ff6c6abac 2021-07-10T09:40:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122ff6c6abac 2021-07-10T09:40:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:33:03 2021 2021-07-10T09:40:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:33:03 2021 2021-07-10T09:40:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:33:03 2021 2021-07-10T09:40:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 66942d87-456e-4d29-98b9-0ee310c0e239 2021-07-10T09:40:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_14 snap 2021-07-10T09:40:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:40:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.export 2021-07-10T09:40:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.export 2021-07-10T09:40:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.export 2021-07-10T09:40:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.export 2021-07-10T09:40:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.export 2021-07-10T09:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.export 2021-07-10T09:40:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_14.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_14.export 2021-07-10T09:40:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_15 2021-07-10T09:40:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_15 snap 2021-07-10T09:40:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2021-07-10T09:40:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2021-07-10T09:40:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2021-07-10T09:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2021-07-10T09:40:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2021-07-10T09:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2021-07-10T09:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2021-07-10T09:40:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2021-07-10T09:40:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_15.status 2021-07-10T09:40:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_15.status 2021-07-10T09:40:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.status 2021-07-10T09:40:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_15.status 2021-07-10T09:40:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_15.status 2021-07-10T09:40:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.status 2021-07-10T09:40:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2021-07-10T09:40:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2021-07-10T09:40:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 161 2021-07-10T09:40:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.status 2021-07-10T09:40:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.status 2021-07-10T09:40:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2021-07-10T09:40:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=161 2021-07-10T09:40:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.status 2021-07-10T09:40:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.status 2021-07-10T09:40:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2021-07-10T09:40:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 161 = '' 2021-07-10T09:40:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.status 2021-07-10T09:40:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.status 2021-07-10T09:40:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2021-07-10T09:40:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 161 = '' 2021-07-10T09:40:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.status 2021-07-10T09:40:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.status 2021-07-10T09:40:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2021-07-10T09:40:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=161 2021-07-10T09:40:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 161 = 161 2021-07-10T09:40:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2021-07-10T09:40:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2021-07-10T09:40:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_15': 2021-07-10T09:40:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12a51a809527 2021-07-10T09:40:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12a51a809527 2021-07-10T09:40:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:33:18 2021 2021-07-10T09:40:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:33:18 2021 2021-07-10T09:40:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:33:18 2021 2021-07-10T09:40:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: f673ef0c-641d-481b-b265-6eedeff68e60 2021-07-10T09:40:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_15 snap 2021-07-10T09:40:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:40:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.export 2021-07-10T09:40:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.export 2021-07-10T09:40:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.export 2021-07-10T09:40:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.export 2021-07-10T09:40:53.425 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:53.425+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:40:53.426 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:53.425+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:40:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.export 2021-07-10T09:40:53.584 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:53.580+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:53.584 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:53.580+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:40:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.export 2021-07-10T09:40:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_15.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_15.export 2021-07-10T09:40:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:40:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_16 2021-07-10T09:40:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_16 snap 2021-07-10T09:40:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:40:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2021-07-10T09:40:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2021-07-10T09:40:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:40:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2021-07-10T09:40:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:40:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:40:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:40:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2021-07-10T09:40:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:40:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:40:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:40:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:40:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:40:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:40:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:40:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:40:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:40:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2021-07-10T09:40:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:40:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:40:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:40:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:40:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:40:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:40:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:40:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:40:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:40:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:40:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:40:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:40:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:40:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:40:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:40:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2021-07-10T09:40:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2021-07-10T09:40:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2021-07-10T09:40:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:40:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:40:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:40:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:40:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:40:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:40:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:40:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:40:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:40:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:40:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:40:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:40:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2021-07-10T09:40:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2021-07-10T09:40:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:40:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:40:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:40:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2021-07-10T09:40:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:40:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:40:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:40:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:40:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:40:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:40:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:40:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:40:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:40:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:40:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:40:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:40:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:40:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:40:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:40:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:40:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:40:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:40:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:40:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:40:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:40:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:40:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:40:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2021-07-10T09:40:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:40:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:40:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:40:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:40:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:40:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:40:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:40:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2021-07-10T09:40:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:40:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:40:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2021-07-10T09:40:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:40:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_16.status 2021-07-10T09:40:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_16.status 2021-07-10T09:40:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.status 2021-07-10T09:40:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_16.status 2021-07-10T09:40:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_16.status 2021-07-10T09:40:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:40:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.status 2021-07-10T09:40:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2021-07-10T09:40:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2021-07-10T09:40:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 164 2021-07-10T09:40:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.status 2021-07-10T09:40:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:40:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.status 2021-07-10T09:40:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2021-07-10T09:40:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:40:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=164 2021-07-10T09:40:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:40:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.status 2021-07-10T09:40:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.status 2021-07-10T09:40:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2021-07-10T09:40:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 164 = '' 2021-07-10T09:40:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:40:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.status 2021-07-10T09:40:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.status 2021-07-10T09:40:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2021-07-10T09:40:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:40:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:40:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:40:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:40:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:40:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 164 = '' 2021-07-10T09:40:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:40:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:40:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.status 2021-07-10T09:40:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.status 2021-07-10T09:40:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2021-07-10T09:40:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:40:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:40:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=164 2021-07-10T09:40:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 164 = 164 2021-07-10T09:40:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2021-07-10T09:40:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:40:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:40:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:40:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2021-07-10T09:40:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_16': 2021-07-10T09:40:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:40:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:40:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:40:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d4d8001f0 2021-07-10T09:40:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d4d8001f0 2021-07-10T09:40:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:40:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:33:32 2021 2021-07-10T09:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:33:32 2021 2021-07-10T09:40:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:33:32 2021 2021-07-10T09:40:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:40:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:40:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:40:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 67f49a84-44f6-4208-ab12-898fd7c3d389 2021-07-10T09:40:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:40:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:40:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_16 snap 2021-07-10T09:40:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:40:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:40:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:40:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:40:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.export 2021-07-10T09:40:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.export 2021-07-10T09:40:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.export 2021-07-10T09:40:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.export 2021-07-10T09:40:59.477 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:59.474+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:40:59.477 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:59.475+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:40:59.478 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:40:59.475+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:40:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.export 2021-07-10T09:40:59.862 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:40:59.861+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:40:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:40:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.export 2021-07-10T09:41:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_16.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_16.export 2021-07-10T09:41:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_17 2021-07-10T09:41:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_17 snap 2021-07-10T09:41:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2021-07-10T09:41:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2021-07-10T09:41:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2021-07-10T09:41:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2021-07-10T09:41:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2021-07-10T09:41:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2021-07-10T09:41:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2021-07-10T09:41:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2021-07-10T09:41:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:41:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:41:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:41:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:41:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:41:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:41:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:41:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:41:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:41:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:41:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:41:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:41:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2021-07-10T09:41:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2021-07-10T09:41:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2021-07-10T09:41:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_17.status 2021-07-10T09:41:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_17.status 2021-07-10T09:41:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.status 2021-07-10T09:41:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_17.status 2021-07-10T09:41:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_17.status 2021-07-10T09:41:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.status 2021-07-10T09:41:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2021-07-10T09:41:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2021-07-10T09:41:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 167 2021-07-10T09:41:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.status 2021-07-10T09:41:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.status 2021-07-10T09:41:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2021-07-10T09:41:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=167 2021-07-10T09:41:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.status 2021-07-10T09:41:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.status 2021-07-10T09:41:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2021-07-10T09:41:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 167 = '' 2021-07-10T09:41:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.status 2021-07-10T09:41:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.status 2021-07-10T09:41:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2021-07-10T09:41:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 167 = '' 2021-07-10T09:41:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.status 2021-07-10T09:41:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.status 2021-07-10T09:41:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2021-07-10T09:41:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=167 2021-07-10T09:41:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 167 = 167 2021-07-10T09:41:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2021-07-10T09:41:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2021-07-10T09:41:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_17': 2021-07-10T09:41:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a4b13fadf4 2021-07-10T09:41:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a4b13fadf4 2021-07-10T09:41:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:33:43 2021 2021-07-10T09:41:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:33:43 2021 2021-07-10T09:41:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:33:43 2021 2021-07-10T09:41:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 8591cd05-0cb5-4297-8136-b3d3c7aa0f66 2021-07-10T09:41:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_17 snap 2021-07-10T09:41:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:41:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.export 2021-07-10T09:41:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.export 2021-07-10T09:41:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.export 2021-07-10T09:41:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.export 2021-07-10T09:41:05.395 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:05.393+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:41:05.395 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:05.394+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2021-07-10T09:41:05.461 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:05.459+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:41:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.export 2021-07-10T09:41:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.export 2021-07-10T09:41:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_17.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_17.export 2021-07-10T09:41:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_18 2021-07-10T09:41:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_18 snap 2021-07-10T09:41:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2021-07-10T09:41:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2021-07-10T09:41:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2021-07-10T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2021-07-10T09:41:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2021-07-10T09:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2021-07-10T09:41:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2021-07-10T09:41:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2021-07-10T09:41:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:41:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:41:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:41:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:41:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:41:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:41:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:41:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:41:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:41:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:41:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:41:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:41:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2021-07-10T09:41:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2021-07-10T09:41:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2021-07-10T09:41:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:41:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:41:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:41:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:41:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:41:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:41:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:41:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:41:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:41:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:41:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:41:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:41:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2021-07-10T09:41:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2021-07-10T09:41:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2021-07-10T09:41:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_18.status 2021-07-10T09:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_18.status 2021-07-10T09:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.status 2021-07-10T09:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_18.status 2021-07-10T09:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_18.status 2021-07-10T09:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.status 2021-07-10T09:41:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2021-07-10T09:41:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2021-07-10T09:41:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 170 2021-07-10T09:41:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.status 2021-07-10T09:41:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.status 2021-07-10T09:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2021-07-10T09:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=170 2021-07-10T09:41:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.status 2021-07-10T09:41:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.status 2021-07-10T09:41:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2021-07-10T09:41:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 170 = '' 2021-07-10T09:41:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.status 2021-07-10T09:41:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.status 2021-07-10T09:41:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2021-07-10T09:41:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 170 = '' 2021-07-10T09:41:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.status 2021-07-10T09:41:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.status 2021-07-10T09:41:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2021-07-10T09:41:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=170 2021-07-10T09:41:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 170 = 170 2021-07-10T09:41:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2021-07-10T09:41:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:10.700 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.824762190921331) > weight (0.25) 2021-07-10T09:41:10.701 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2021-07-10T09:41:10.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2021-07-10T09:41:10.702 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2021-07-10T09:41:10.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi105.stderr:2021-07-10T09:41:10.701+0000 7f797338f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 46439) UID: 1000 2021-07-10T09:41:10.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:41:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2021-07-10T09:41:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_18': 2021-07-10T09:41:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f740203ee 2021-07-10T09:41:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f740203ee 2021-07-10T09:41:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:33:55 2021 2021-07-10T09:41:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:33:55 2021 2021-07-10T09:41:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:33:55 2021 2021-07-10T09:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 1916d838-e47d-4e6b-a0f7-ba55f103b403 2021-07-10T09:41:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_18 snap 2021-07-10T09:41:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:41:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.export 2021-07-10T09:41:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.export 2021-07-10T09:41:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.export 2021-07-10T09:41:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.export 2021-07-10T09:41:11.494 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:11.493+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:41:11.495 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:11.494+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:41:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.export 2021-07-10T09:41:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.export 2021-07-10T09:41:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_18.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_18.export 2021-07-10T09:41:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_19 2021-07-10T09:41:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_19 snap 2021-07-10T09:41:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2021-07-10T09:41:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2021-07-10T09:41:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2021-07-10T09:41:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2021-07-10T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2021-07-10T09:41:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2021-07-10T09:41:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2021-07-10T09:41:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2021-07-10T09:41:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_19.status 2021-07-10T09:41:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_19.status 2021-07-10T09:41:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.status 2021-07-10T09:41:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_19.status 2021-07-10T09:41:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_19.status 2021-07-10T09:41:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.status 2021-07-10T09:41:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2021-07-10T09:41:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2021-07-10T09:41:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 173 2021-07-10T09:41:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.status 2021-07-10T09:41:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.status 2021-07-10T09:41:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2021-07-10T09:41:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=173 2021-07-10T09:41:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.status 2021-07-10T09:41:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.status 2021-07-10T09:41:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2021-07-10T09:41:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 173 = '' 2021-07-10T09:41:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.status 2021-07-10T09:41:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.status 2021-07-10T09:41:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2021-07-10T09:41:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 173 = '' 2021-07-10T09:41:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:15.702 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2021-07-10T09:41:15.703 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:41:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:41:15.704 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2021-07-10T09:41:15.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2021-07-10T09:41:15.704 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2021-07-10T09:41:15.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2021-07-10T09:41:15.708 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2021-07-10T09:41:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.status 2021-07-10T09:41:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.status 2021-07-10T09:41:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2021-07-10T09:41:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=173 2021-07-10T09:41:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 173 = 173 2021-07-10T09:41:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2021-07-10T09:41:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2021-07-10T09:41:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_19': 2021-07-10T09:41:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12a545685d7a 2021-07-10T09:41:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12a545685d7a 2021-07-10T09:41:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:34:08 2021 2021-07-10T09:41:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:34:08 2021 2021-07-10T09:41:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:34:08 2021 2021-07-10T09:41:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 442bb14f-6de4-4753-b4e0-c19c49b51af9 2021-07-10T09:41:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_19 snap 2021-07-10T09:41:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:41:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.export 2021-07-10T09:41:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.export 2021-07-10T09:41:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.export 2021-07-10T09:41:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.export 2021-07-10T09:41:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.export 2021-07-10T09:41:17.700 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:17.697+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:41:17.700 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:17.698+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:41:17.700 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:17.699+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:41:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.export 2021-07-10T09:41:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_19.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_19.export 2021-07-10T09:41:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_20 2021-07-10T09:41:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_20 snap 2021-07-10T09:41:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2021-07-10T09:41:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2021-07-10T09:41:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2021-07-10T09:41:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2021-07-10T09:41:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2021-07-10T09:41:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2021-07-10T09:41:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2021-07-10T09:41:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2021-07-10T09:41:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:41:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:41:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:41:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:41:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:41:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:41:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:41:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:41:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:41:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:41:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:41:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:41:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2021-07-10T09:41:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2021-07-10T09:41:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2021-07-10T09:41:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:41:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:41:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:41:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:41:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:41:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:41:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:41:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:41:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:41:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:41:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:41:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:41:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2021-07-10T09:41:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2021-07-10T09:41:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2021-07-10T09:41:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:41:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:41:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:41:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:41:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:41:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:41:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:41:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:41:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:41:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:41:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:41:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:41:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:41:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:41:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:41:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:41:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:41:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:41:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:41:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2021-07-10T09:41:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2021-07-10T09:41:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2021-07-10T09:41:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_20.status 2021-07-10T09:41:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_20.status 2021-07-10T09:41:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.status 2021-07-10T09:41:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_20.status 2021-07-10T09:41:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_20.status 2021-07-10T09:41:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.status 2021-07-10T09:41:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2021-07-10T09:41:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2021-07-10T09:41:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 176 2021-07-10T09:41:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.status 2021-07-10T09:41:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.status 2021-07-10T09:41:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2021-07-10T09:41:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=176 2021-07-10T09:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.status 2021-07-10T09:41:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.status 2021-07-10T09:41:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2021-07-10T09:41:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 176 = '' 2021-07-10T09:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.status 2021-07-10T09:41:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.status 2021-07-10T09:41:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2021-07-10T09:41:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 176 = '' 2021-07-10T09:41:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.status 2021-07-10T09:41:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.status 2021-07-10T09:41:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2021-07-10T09:41:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=176 2021-07-10T09:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 176 = 176 2021-07-10T09:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2021-07-10T09:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2021-07-10T09:41:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_20': 2021-07-10T09:41:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d46b06123b 2021-07-10T09:41:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d46b06123b 2021-07-10T09:41:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:34:14 2021 2021-07-10T09:41:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:34:14 2021 2021-07-10T09:41:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:34:14 2021 2021-07-10T09:41:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 14c6445c-0ffb-47f4-9ad5-108e6a8a92fa 2021-07-10T09:41:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_20 snap 2021-07-10T09:41:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:41:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.export 2021-07-10T09:41:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.export 2021-07-10T09:41:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.export 2021-07-10T09:41:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.export 2021-07-10T09:41:23.364 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:23.363+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:41:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.export 2021-07-10T09:41:23.704 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:41:23.703+0000 7f72c608f700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2021-07-10T09:41:23.704 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:41:23.703+0000 7f72cc09b700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 7 2021-07-10T09:41:23.709 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:41:23.705+0000 7f72c9095700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2021-07-10T09:41:23.776 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:41:23.775+0000 7f72c8093700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:41:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.export 2021-07-10T09:41:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_20.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_20.export 2021-07-10T09:41:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_21 2021-07-10T09:41:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_21 snap 2021-07-10T09:41:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2021-07-10T09:41:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2021-07-10T09:41:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2021-07-10T09:41:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2021-07-10T09:41:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2021-07-10T09:41:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2021-07-10T09:41:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2021-07-10T09:41:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2021-07-10T09:41:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:41:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:41:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:41:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:41:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:41:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:41:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:41:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:41:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:41:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:41:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2021-07-10T09:41:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2021-07-10T09:41:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2021-07-10T09:41:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_21.status 2021-07-10T09:41:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_21.status 2021-07-10T09:41:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.status 2021-07-10T09:41:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_21.status 2021-07-10T09:41:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_21.status 2021-07-10T09:41:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.status 2021-07-10T09:41:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2021-07-10T09:41:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2021-07-10T09:41:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 179 2021-07-10T09:41:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.status 2021-07-10T09:41:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.status 2021-07-10T09:41:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2021-07-10T09:41:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=179 2021-07-10T09:41:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.status 2021-07-10T09:41:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.status 2021-07-10T09:41:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2021-07-10T09:41:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 179 = '' 2021-07-10T09:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.status 2021-07-10T09:41:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.status 2021-07-10T09:41:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2021-07-10T09:41:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 179 = '' 2021-07-10T09:41:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.status 2021-07-10T09:41:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.status 2021-07-10T09:41:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2021-07-10T09:41:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=179 2021-07-10T09:41:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 179 = 179 2021-07-10T09:41:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2021-07-10T09:41:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2021-07-10T09:41:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_21': 2021-07-10T09:41:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a4e0794af6 2021-07-10T09:41:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a4e0794af6 2021-07-10T09:41:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:34:27 2021 2021-07-10T09:41:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:34:27 2021 2021-07-10T09:41:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:34:27 2021 2021-07-10T09:41:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: fc5e0363-04c0-422c-abf7-b7c659415f70 2021-07-10T09:41:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_21 snap 2021-07-10T09:41:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:41:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.export 2021-07-10T09:41:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.export 2021-07-10T09:41:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.export 2021-07-10T09:41:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.export 2021-07-10T09:41:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.export 2021-07-10T09:41:29.723 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:41:29.722+0000 7f4228176700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:41:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.export 2021-07-10T09:41:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_21.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_21.export 2021-07-10T09:41:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_22 2021-07-10T09:41:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_22 snap 2021-07-10T09:41:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2021-07-10T09:41:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2021-07-10T09:41:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2021-07-10T09:41:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2021-07-10T09:41:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2021-07-10T09:41:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2021-07-10T09:41:31.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2021-07-10T09:41:31.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:31.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2021-07-10T09:41:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:41:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:41:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:41:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:41:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:41:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:41:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:41:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:41:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2021-07-10T09:41:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2021-07-10T09:41:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2021-07-10T09:41:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:41:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:41:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:41:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:41:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:41:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:41:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:41:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:41:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:41:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:41:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2021-07-10T09:41:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2021-07-10T09:41:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2021-07-10T09:41:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_22.status 2021-07-10T09:41:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_22.status 2021-07-10T09:41:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.status 2021-07-10T09:41:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_22.status 2021-07-10T09:41:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_22.status 2021-07-10T09:41:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.status 2021-07-10T09:41:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2021-07-10T09:41:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2021-07-10T09:41:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 182 2021-07-10T09:41:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.status 2021-07-10T09:41:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.status 2021-07-10T09:41:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2021-07-10T09:41:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=182 2021-07-10T09:41:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.status 2021-07-10T09:41:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.status 2021-07-10T09:41:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2021-07-10T09:41:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 182 = '' 2021-07-10T09:41:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.status 2021-07-10T09:41:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.status 2021-07-10T09:41:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2021-07-10T09:41:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 182 = '' 2021-07-10T09:41:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.status 2021-07-10T09:41:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.status 2021-07-10T09:41:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2021-07-10T09:41:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=182 2021-07-10T09:41:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 182 = 182 2021-07-10T09:41:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2021-07-10T09:41:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2021-07-10T09:41:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_22': 2021-07-10T09:41:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f60b4cf2e 2021-07-10T09:41:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f60b4cf2e 2021-07-10T09:41:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:34:37 2021 2021-07-10T09:41:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:34:37 2021 2021-07-10T09:41:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:34:37 2021 2021-07-10T09:41:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 5ddfe965-8561-48dc-be37-e53b6b452406 2021-07-10T09:41:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_22 snap 2021-07-10T09:41:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:41:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.export 2021-07-10T09:41:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.export 2021-07-10T09:41:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.export 2021-07-10T09:41:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.export 2021-07-10T09:41:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.export 2021-07-10T09:41:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.export 2021-07-10T09:41:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_22.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_22.export 2021-07-10T09:41:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_23 2021-07-10T09:41:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_23 snap 2021-07-10T09:41:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2021-07-10T09:41:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2021-07-10T09:41:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2021-07-10T09:41:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2021-07-10T09:41:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2021-07-10T09:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2021-07-10T09:41:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2021-07-10T09:41:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2021-07-10T09:41:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_23.status 2021-07-10T09:41:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_23.status 2021-07-10T09:41:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.status 2021-07-10T09:41:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_23.status 2021-07-10T09:41:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_23.status 2021-07-10T09:41:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.status 2021-07-10T09:41:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2021-07-10T09:41:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2021-07-10T09:41:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 185 2021-07-10T09:41:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.status 2021-07-10T09:41:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.status 2021-07-10T09:41:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2021-07-10T09:41:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=185 2021-07-10T09:41:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.status 2021-07-10T09:41:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.status 2021-07-10T09:41:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2021-07-10T09:41:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 185 = '' 2021-07-10T09:41:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.status 2021-07-10T09:41:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.status 2021-07-10T09:41:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2021-07-10T09:41:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 185 = '' 2021-07-10T09:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:39.998 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:39.997+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:41:39.998 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:39.997+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:41:39.998 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:39.997+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:41:39.999 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:39.997+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:41:39.999 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:39.997+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:41:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.status 2021-07-10T09:41:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.status 2021-07-10T09:41:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2021-07-10T09:41:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=185 2021-07-10T09:41:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 185 = 185 2021-07-10T09:41:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2021-07-10T09:41:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2021-07-10T09:41:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_23': 2021-07-10T09:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 3 2021-07-10T09:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12a5a6348553 2021-07-10T09:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12a5a6348553 2021-07-10T09:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, operations, non-primary 2021-07-10T09:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: snap-trash 2021-07-10T09:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:34:43 2021 2021-07-10T09:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:34:43 2021 2021-07-10T09:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:34:43 2021 2021-07-10T09:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 3b64483a-8dc2-4949-8758-67a48f2f7e80 2021-07-10T09:41:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_23 snap 2021-07-10T09:41:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:41:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.export 2021-07-10T09:41:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.export 2021-07-10T09:41:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.export 2021-07-10T09:41:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.export 2021-07-10T09:41:41.395 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:41.393+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:41:41.395 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:41.393+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:41:41.396 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:41.393+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2021-07-10T09:41:41.396 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:41:41.394+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:41:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.export 2021-07-10T09:41:41.610 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:41.608+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:41:41.611 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:41.609+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:41:41.690 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:41.689+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:41:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.export 2021-07-10T09:41:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_23.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_23.export 2021-07-10T09:41:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_24 2021-07-10T09:41:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_24 snap 2021-07-10T09:41:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2021-07-10T09:41:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2021-07-10T09:41:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2021-07-10T09:41:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2021-07-10T09:41:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2021-07-10T09:41:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2021-07-10T09:41:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2021-07-10T09:41:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2021-07-10T09:41:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:41:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:41:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:41:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:41:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:41:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:41:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:41:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:41:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:41:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:41:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:41:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:41:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:41:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:41:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:41:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2021-07-10T09:41:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2021-07-10T09:41:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2021-07-10T09:41:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:41:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:41:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:41:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:41:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:41:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:41:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:41:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:41:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:41:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:41:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:41:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2021-07-10T09:41:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2021-07-10T09:41:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:41:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:41:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2021-07-10T09:41:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:41:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:41:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:41:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:41:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:41:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:41:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:41:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:41:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:41:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:41:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:41:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:41:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:41:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:41:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:41:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:41:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:41:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:41:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:41:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2021-07-10T09:41:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2021-07-10T09:41:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2021-07-10T09:41:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_24.status 2021-07-10T09:41:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_24.status 2021-07-10T09:41:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.status 2021-07-10T09:41:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_24.status 2021-07-10T09:41:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_24.status 2021-07-10T09:41:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2021-07-10T09:41:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2021-07-10T09:41:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 188 2021-07-10T09:41:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.status 2021-07-10T09:41:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.status 2021-07-10T09:41:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2021-07-10T09:41:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=188 2021-07-10T09:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2021-07-10T09:41:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=91 2021-07-10T09:41:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 188 = 91 2021-07-10T09:41:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2021-07-10T09:41:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=91 2021-07-10T09:41:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 188 = 91 2021-07-10T09:41:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2021-07-10T09:41:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 188 = '' 2021-07-10T09:41:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:41:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.status 2021-07-10T09:41:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2021-07-10T09:41:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:47.909 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5178997945180073) > weight (0.25) 2021-07-10T09:41:47.910 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9950069138800572) > weight (0.25) 2021-07-10T09:41:47.910 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2021-07-10T09:41:47.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2021-07-10T09:41:47.911 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2021-07-10T09:41:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=188 2021-07-10T09:41:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 188 = 188 2021-07-10T09:41:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2021-07-10T09:41:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:47.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:2021-07-10T09:41:47.911+0000 7fc5eec9f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 44902) UID: 1000 2021-07-10T09:41:48.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:41:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2021-07-10T09:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_24': 2021-07-10T09:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d4b5d45bb1 2021-07-10T09:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d4b5d45bb1 2021-07-10T09:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:34:54 2021 2021-07-10T09:41:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:34:54 2021 2021-07-10T09:41:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:34:54 2021 2021-07-10T09:41:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 740af2ea-fdc1-43cb-a7f2-44ce6928c38c 2021-07-10T09:41:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_24 snap 2021-07-10T09:41:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:41:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.export 2021-07-10T09:41:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.export 2021-07-10T09:41:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.export 2021-07-10T09:41:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.export 2021-07-10T09:41:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.export 2021-07-10T09:41:48.744 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:48.743+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:41:48.745 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:48.744+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:41:48.747 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:48.745+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:41:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.export 2021-07-10T09:41:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_24.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_24.export 2021-07-10T09:41:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_25 2021-07-10T09:41:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_25 snap 2021-07-10T09:41:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:48.911 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2021-07-10T09:41:48.912 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:41:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:41:48.913 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2021-07-10T09:41:48.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2021-07-10T09:41:48.913 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2021-07-10T09:41:48.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2021-07-10T09:41:48.919 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2021-07-10T09:41:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2021-07-10T09:41:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2021-07-10T09:41:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2021-07-10T09:41:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2021-07-10T09:41:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2021-07-10T09:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2021-07-10T09:41:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2021-07-10T09:41:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2021-07-10T09:41:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_25.status 2021-07-10T09:41:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_25.status 2021-07-10T09:41:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.status 2021-07-10T09:41:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_25.status 2021-07-10T09:41:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_25.status 2021-07-10T09:41:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.status 2021-07-10T09:41:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2021-07-10T09:41:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2021-07-10T09:41:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 191 2021-07-10T09:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.status 2021-07-10T09:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.status 2021-07-10T09:41:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2021-07-10T09:41:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=191 2021-07-10T09:41:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.status 2021-07-10T09:41:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.status 2021-07-10T09:41:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2021-07-10T09:41:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 191 = '' 2021-07-10T09:41:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.status 2021-07-10T09:41:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.status 2021-07-10T09:41:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2021-07-10T09:41:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 191 = '' 2021-07-10T09:41:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:52.877 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:41:52.875+0000 7f69e7730700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:41:52.878 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:41:52.876+0000 7f69e472a700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:41:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.status 2021-07-10T09:41:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.status 2021-07-10T09:41:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2021-07-10T09:41:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=191 2021-07-10T09:41:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 191 = 191 2021-07-10T09:41:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2021-07-10T09:41:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2021-07-10T09:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_25': 2021-07-10T09:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:41:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e99c7369f2 2021-07-10T09:41:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e99c7369f2 2021-07-10T09:41:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:41:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:41:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:41:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:41:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:35:00 2021 2021-07-10T09:41:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:35:00 2021 2021-07-10T09:41:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:35:00 2021 2021-07-10T09:41:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:41:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:41:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:41:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 5cb0b35c-161d-4b7f-9f8b-2537969f233f 2021-07-10T09:41:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:41:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_25 snap 2021-07-10T09:41:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:41:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.export 2021-07-10T09:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.export 2021-07-10T09:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.export 2021-07-10T09:41:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.export 2021-07-10T09:41:54.516 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:41:54.514+0000 7f511128e700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:41:54.516 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:41:54.514+0000 7f5113a93700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:41:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.export 2021-07-10T09:41:54.710 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:41:54.709+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:41:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:41:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.export 2021-07-10T09:41:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_25.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_25.export 2021-07-10T09:41:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:41:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_26 2021-07-10T09:41:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_26 snap 2021-07-10T09:41:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:41:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2021-07-10T09:41:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2021-07-10T09:41:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:41:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:41:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2021-07-10T09:41:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:41:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:41:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:41:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2021-07-10T09:41:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:41:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:41:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:41:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:41:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:41:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:41:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:41:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:41:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:41:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:41:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2021-07-10T09:41:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:41:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:41:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:41:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:41:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:41:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:41:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:41:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:41:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:41:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:41:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:41:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:41:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:41:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:41:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:41:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:41:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:41:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:41:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:41:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:41:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:41:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:41:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:41:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:41:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:41:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:41:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:41:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:41:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2021-07-10T09:41:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:41:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:41:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:41:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:41:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:41:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:41:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:41:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2021-07-10T09:41:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:41:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:41:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2021-07-10T09:41:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:41:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_26.status 2021-07-10T09:41:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_26.status 2021-07-10T09:41:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.status 2021-07-10T09:41:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_26.status 2021-07-10T09:41:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_26.status 2021-07-10T09:41:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:41:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.status 2021-07-10T09:41:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2021-07-10T09:41:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2021-07-10T09:41:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 194 2021-07-10T09:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.status 2021-07-10T09:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.status 2021-07-10T09:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2021-07-10T09:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:41:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=194 2021-07-10T09:41:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:41:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.status 2021-07-10T09:41:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.status 2021-07-10T09:41:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2021-07-10T09:41:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 194 = '' 2021-07-10T09:41:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:41:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.status 2021-07-10T09:41:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.status 2021-07-10T09:41:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2021-07-10T09:41:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:41:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:41:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:41:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 194 = '' 2021-07-10T09:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:41:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.status 2021-07-10T09:41:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.status 2021-07-10T09:41:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2021-07-10T09:41:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:41:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:41:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=194 2021-07-10T09:41:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 194 = 194 2021-07-10T09:41:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:41:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2021-07-10T09:41:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:41:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:41:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:41:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:41:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:41:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:41:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2021-07-10T09:42:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_26': 2021-07-10T09:42:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e9b0f6eff5 2021-07-10T09:42:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e9b0f6eff5 2021-07-10T09:42:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:35:09 2021 2021-07-10T09:42:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:35:09 2021 2021-07-10T09:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:35:09 2021 2021-07-10T09:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 9c01f8cb-62f8-400b-bc91-2d0fe03eb4d3 2021-07-10T09:42:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_26 snap 2021-07-10T09:42:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:42:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.export 2021-07-10T09:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.export 2021-07-10T09:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.export 2021-07-10T09:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.export 2021-07-10T09:42:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.export 2021-07-10T09:42:00.699 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:42:00.698+0000 7f72c7892700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2021-07-10T09:42:00.700 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:42:00.698+0000 7f72c6890700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2021-07-10T09:42:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.export 2021-07-10T09:42:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_26.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_26.export 2021-07-10T09:42:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_27 2021-07-10T09:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_27 snap 2021-07-10T09:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2021-07-10T09:42:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2021-07-10T09:42:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2021-07-10T09:42:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2021-07-10T09:42:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2021-07-10T09:42:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2021-07-10T09:42:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2021-07-10T09:42:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2021-07-10T09:42:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_27.status 2021-07-10T09:42:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_27.status 2021-07-10T09:42:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.status 2021-07-10T09:42:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_27.status 2021-07-10T09:42:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_27.status 2021-07-10T09:42:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.status 2021-07-10T09:42:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2021-07-10T09:42:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2021-07-10T09:42:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 197 2021-07-10T09:42:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.status 2021-07-10T09:42:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.status 2021-07-10T09:42:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2021-07-10T09:42:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=197 2021-07-10T09:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.status 2021-07-10T09:42:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.status 2021-07-10T09:42:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2021-07-10T09:42:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 197 = '' 2021-07-10T09:42:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.status 2021-07-10T09:42:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.status 2021-07-10T09:42:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2021-07-10T09:42:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 197 = '' 2021-07-10T09:42:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.status 2021-07-10T09:42:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.status 2021-07-10T09:42:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2021-07-10T09:42:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=197 2021-07-10T09:42:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 197 = 197 2021-07-10T09:42:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2021-07-10T09:42:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:05.964 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:05.960+0000 7f25844e6700 -1 reset not still connected to 0x55571456c410 2021-07-10T09:42:05.964 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:05.961+0000 7f25844e6700 -1 reset not still connected to 0x5557312535f0 2021-07-10T09:42:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2021-07-10T09:42:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_27': 2021-07-10T09:42:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e9e6b11969 2021-07-10T09:42:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e9e6b11969 2021-07-10T09:42:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:35:17 2021 2021-07-10T09:42:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:35:17 2021 2021-07-10T09:42:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:35:17 2021 2021-07-10T09:42:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 4edf81d8-ee1c-4b77-9936-60b91c73e9ae 2021-07-10T09:42:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_27 snap 2021-07-10T09:42:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:42:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.export 2021-07-10T09:42:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.export 2021-07-10T09:42:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.export 2021-07-10T09:42:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.export 2021-07-10T09:42:06.457 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:42:06.455+0000 7f5113a93700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:42:06.509 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:06.507+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:42:06.510 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:06.508+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2021-07-10T09:42:06.511 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:06.510+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2021-07-10T09:42:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.export 2021-07-10T09:42:06.624 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:42:06.622+0000 7f4227975700 -1 bdev(0x55a447aa0000 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2021-07-10T09:42:06.782 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:06.780+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:42:06.783 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:06.782+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:42:06.783 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:06.782+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:42:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.export 2021-07-10T09:42:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_27.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_27.export 2021-07-10T09:42:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_28 2021-07-10T09:42:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_28 snap 2021-07-10T09:42:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2021-07-10T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2021-07-10T09:42:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2021-07-10T09:42:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2021-07-10T09:42:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2021-07-10T09:42:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2021-07-10T09:42:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2021-07-10T09:42:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2021-07-10T09:42:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_28.status 2021-07-10T09:42:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_28.status 2021-07-10T09:42:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.status 2021-07-10T09:42:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_28.status 2021-07-10T09:42:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_28.status 2021-07-10T09:42:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.status 2021-07-10T09:42:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2021-07-10T09:42:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2021-07-10T09:42:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 200 2021-07-10T09:42:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.status 2021-07-10T09:42:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.status 2021-07-10T09:42:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2021-07-10T09:42:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=200 2021-07-10T09:42:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.status 2021-07-10T09:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.status 2021-07-10T09:42:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2021-07-10T09:42:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 200 = '' 2021-07-10T09:42:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.status 2021-07-10T09:42:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.status 2021-07-10T09:42:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2021-07-10T09:42:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 200 = '' 2021-07-10T09:42:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:11.107 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:42:11.105+0000 7f72c508d700 -1 bdev(0x55dd1fba8000 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2021-07-10T09:42:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.status 2021-07-10T09:42:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.status 2021-07-10T09:42:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2021-07-10T09:42:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=200 2021-07-10T09:42:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 200 = 200 2021-07-10T09:42:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2021-07-10T09:42:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2021-07-10T09:42:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_28': 2021-07-10T09:42:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e91441ffe7 2021-07-10T09:42:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e91441ffe7 2021-07-10T09:42:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:35:29 2021 2021-07-10T09:42:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:35:29 2021 2021-07-10T09:42:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:35:29 2021 2021-07-10T09:42:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 7862ad14-d5e1-47bf-bcff-5b59529adf2c 2021-07-10T09:42:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_28 snap 2021-07-10T09:42:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:42:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.export 2021-07-10T09:42:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.export 2021-07-10T09:42:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.export 2021-07-10T09:42:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.export 2021-07-10T09:42:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.export 2021-07-10T09:42:12.813 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:12.812+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:42:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.export 2021-07-10T09:42:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_28.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_28.export 2021-07-10T09:42:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_29 2021-07-10T09:42:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_29 snap 2021-07-10T09:42:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2021-07-10T09:42:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2021-07-10T09:42:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2021-07-10T09:42:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2021-07-10T09:42:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2021-07-10T09:42:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2021-07-10T09:42:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2021-07-10T09:42:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2021-07-10T09:42:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:42:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:42:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:42:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:42:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:42:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:42:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:42:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:42:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:42:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:42:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:42:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:42:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2021-07-10T09:42:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2021-07-10T09:42:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2021-07-10T09:42:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:42:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:42:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:42:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:42:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:42:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:42:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:42:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:42:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:42:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:42:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:42:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:42:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:42:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:42:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:42:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:42:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2021-07-10T09:42:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2021-07-10T09:42:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2021-07-10T09:42:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_29.status 2021-07-10T09:42:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_29.status 2021-07-10T09:42:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.status 2021-07-10T09:42:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_29.status 2021-07-10T09:42:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_29.status 2021-07-10T09:42:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.status 2021-07-10T09:42:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2021-07-10T09:42:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2021-07-10T09:42:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 203 2021-07-10T09:42:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.status 2021-07-10T09:42:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.status 2021-07-10T09:42:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2021-07-10T09:42:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=203 2021-07-10T09:42:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.status 2021-07-10T09:42:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.status 2021-07-10T09:42:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2021-07-10T09:42:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 203 = '' 2021-07-10T09:42:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.status 2021-07-10T09:42:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.status 2021-07-10T09:42:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2021-07-10T09:42:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 203 = '' 2021-07-10T09:42:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.status 2021-07-10T09:42:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.status 2021-07-10T09:42:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2021-07-10T09:42:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=203 2021-07-10T09:42:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 203 = 203 2021-07-10T09:42:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2021-07-10T09:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2021-07-10T09:42:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_29': 2021-07-10T09:42:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a4a246f7f6 2021-07-10T09:42:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a4a246f7f6 2021-07-10T09:42:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:35:40 2021 2021-07-10T09:42:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:35:40 2021 2021-07-10T09:42:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:35:40 2021 2021-07-10T09:42:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: ada249ac-bc19-4ece-a4a5-271489776fc6 2021-07-10T09:42:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_29 snap 2021-07-10T09:42:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:42:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.export 2021-07-10T09:42:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.export 2021-07-10T09:42:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.export 2021-07-10T09:42:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.export 2021-07-10T09:42:18.531 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:18.507+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:42:18.531 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:18.507+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:42:18.532 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:18.507+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:42:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.export 2021-07-10T09:42:18.748 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:18.746+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:42:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.export 2021-07-10T09:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_29.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_29.export 2021-07-10T09:42:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_30 2021-07-10T09:42:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_30 snap 2021-07-10T09:42:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2021-07-10T09:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2021-07-10T09:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2021-07-10T09:42:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2021-07-10T09:42:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2021-07-10T09:42:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2021-07-10T09:42:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2021-07-10T09:42:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2021-07-10T09:42:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:42:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:42:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:42:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:42:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:42:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:42:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:42:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:42:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:42:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:42:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:42:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:42:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2021-07-10T09:42:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2021-07-10T09:42:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2021-07-10T09:42:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:42:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:42:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:42:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:42:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:42:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:42:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:42:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:42:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:42:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:42:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:42:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:42:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:42:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:42:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:42:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:42:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2021-07-10T09:42:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2021-07-10T09:42:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2021-07-10T09:42:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_30.status 2021-07-10T09:42:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_30.status 2021-07-10T09:42:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.status 2021-07-10T09:42:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_30.status 2021-07-10T09:42:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_30.status 2021-07-10T09:42:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.status 2021-07-10T09:42:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2021-07-10T09:42:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2021-07-10T09:42:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 206 2021-07-10T09:42:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.status 2021-07-10T09:42:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.status 2021-07-10T09:42:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2021-07-10T09:42:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=206 2021-07-10T09:42:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.status 2021-07-10T09:42:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.status 2021-07-10T09:42:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2021-07-10T09:42:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 206 = '' 2021-07-10T09:42:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.status 2021-07-10T09:42:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.status 2021-07-10T09:42:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2021-07-10T09:42:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 206 = '' 2021-07-10T09:42:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.status 2021-07-10T09:42:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.status 2021-07-10T09:42:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2021-07-10T09:42:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=206 2021-07-10T09:42:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 206 = 206 2021-07-10T09:42:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2021-07-10T09:42:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2021-07-10T09:42:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_30': 2021-07-10T09:42:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f93b4eb86 2021-07-10T09:42:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f93b4eb86 2021-07-10T09:42:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:35:48 2021 2021-07-10T09:42:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:35:48 2021 2021-07-10T09:42:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:35:48 2021 2021-07-10T09:42:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 4482cfc2-819d-41e2-88b5-f7ddc3348b33 2021-07-10T09:42:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_30 snap 2021-07-10T09:42:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:42:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.export 2021-07-10T09:42:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.export 2021-07-10T09:42:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.export 2021-07-10T09:42:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.export 2021-07-10T09:42:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.export 2021-07-10T09:42:24.822 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:24.820+0000 7f3175863700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:42:24.823 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:24.821+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:42:24.823 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:24.822+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:42:24.823 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:24.822+0000 7f317a86d700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:42:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.export 2021-07-10T09:42:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_30.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_30.export 2021-07-10T09:42:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_31 2021-07-10T09:42:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_31 snap 2021-07-10T09:42:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2021-07-10T09:42:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2021-07-10T09:42:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2021-07-10T09:42:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2021-07-10T09:42:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2021-07-10T09:42:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2021-07-10T09:42:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2021-07-10T09:42:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2021-07-10T09:42:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:42:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:42:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:42:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:42:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:42:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:42:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:42:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:42:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:42:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:42:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:42:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:42:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2021-07-10T09:42:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2021-07-10T09:42:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2021-07-10T09:42:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:42:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:42:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:42:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:42:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:42:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:42:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:42:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:42:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:42:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:42:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:42:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:42:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:42:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:42:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:42:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:42:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:42:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2021-07-10T09:42:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2021-07-10T09:42:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2021-07-10T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_31.status 2021-07-10T09:42:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_31.status 2021-07-10T09:42:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.status 2021-07-10T09:42:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_31.status 2021-07-10T09:42:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_31.status 2021-07-10T09:42:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.status 2021-07-10T09:42:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2021-07-10T09:42:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2021-07-10T09:42:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 209 2021-07-10T09:42:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.status 2021-07-10T09:42:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.status 2021-07-10T09:42:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2021-07-10T09:42:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=209 2021-07-10T09:42:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.status 2021-07-10T09:42:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.status 2021-07-10T09:42:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2021-07-10T09:42:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 209 = '' 2021-07-10T09:42:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.status 2021-07-10T09:42:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.status 2021-07-10T09:42:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2021-07-10T09:42:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 209 = '' 2021-07-10T09:42:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:29.062 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:42:29.061+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:42:29.066 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:42:29.063+0000 7f69e3f29700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2021-07-10T09:42:29.067 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:42:29.063+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2021-07-10T09:42:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.status 2021-07-10T09:42:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.status 2021-07-10T09:42:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2021-07-10T09:42:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=209 2021-07-10T09:42:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 209 = 209 2021-07-10T09:42:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2021-07-10T09:42:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2021-07-10T09:42:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_31': 2021-07-10T09:42:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d47777a9a 2021-07-10T09:42:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d47777a9a 2021-07-10T09:42:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:35:55 2021 2021-07-10T09:42:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:35:55 2021 2021-07-10T09:42:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:35:55 2021 2021-07-10T09:42:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: a31d22e5-24f6-4da6-a63a-b39ee89432cb 2021-07-10T09:42:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_31 snap 2021-07-10T09:42:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:42:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.export 2021-07-10T09:42:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.export 2021-07-10T09:42:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.export 2021-07-10T09:42:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.export 2021-07-10T09:42:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.export 2021-07-10T09:42:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.export 2021-07-10T09:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_31.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_31.export 2021-07-10T09:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_32 2021-07-10T09:42:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_32 snap 2021-07-10T09:42:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2021-07-10T09:42:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2021-07-10T09:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2021-07-10T09:42:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2021-07-10T09:42:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2021-07-10T09:42:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2021-07-10T09:42:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2021-07-10T09:42:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2021-07-10T09:42:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_32.status 2021-07-10T09:42:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_32.status 2021-07-10T09:42:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.status 2021-07-10T09:42:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_32.status 2021-07-10T09:42:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_32.status 2021-07-10T09:42:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.status 2021-07-10T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2021-07-10T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2021-07-10T09:42:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 212 2021-07-10T09:42:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.status 2021-07-10T09:42:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.status 2021-07-10T09:42:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2021-07-10T09:42:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=212 2021-07-10T09:42:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.status 2021-07-10T09:42:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.status 2021-07-10T09:42:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2021-07-10T09:42:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 212 = '' 2021-07-10T09:42:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.status 2021-07-10T09:42:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.status 2021-07-10T09:42:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2021-07-10T09:42:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 212 = '' 2021-07-10T09:42:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.status 2021-07-10T09:42:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.status 2021-07-10T09:42:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2021-07-10T09:42:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=212 2021-07-10T09:42:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 212 = 212 2021-07-10T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2021-07-10T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2021-07-10T09:42:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_32': 2021-07-10T09:42:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e9ded6404d 2021-07-10T09:42:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e9ded6404d 2021-07-10T09:42:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:36:02 2021 2021-07-10T09:42:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:36:02 2021 2021-07-10T09:42:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:36:02 2021 2021-07-10T09:42:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: ede0a593-7e45-4306-b6e7-8e106a67c5c9 2021-07-10T09:42:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_32 snap 2021-07-10T09:42:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:42:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.export 2021-07-10T09:42:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.export 2021-07-10T09:42:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.export 2021-07-10T09:42:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.export 2021-07-10T09:42:36.484 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:42:36.483+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2021-07-10T09:42:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.export 2021-07-10T09:42:36.780 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:36.779+0000 7f317a86d700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:42:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.export 2021-07-10T09:42:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_32.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_32.export 2021-07-10T09:42:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_33 2021-07-10T09:42:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_33 snap 2021-07-10T09:42:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2021-07-10T09:42:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2021-07-10T09:42:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2021-07-10T09:42:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2021-07-10T09:42:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2021-07-10T09:42:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2021-07-10T09:42:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2021-07-10T09:42:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2021-07-10T09:42:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:42:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:42:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:42:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:42:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:42:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:42:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:42:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:42:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:42:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:42:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:42:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:42:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2021-07-10T09:42:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2021-07-10T09:42:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2021-07-10T09:42:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_33.status 2021-07-10T09:42:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_33.status 2021-07-10T09:42:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.status 2021-07-10T09:42:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_33.status 2021-07-10T09:42:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_33.status 2021-07-10T09:42:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.status 2021-07-10T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2021-07-10T09:42:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2021-07-10T09:42:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 215 2021-07-10T09:42:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.status 2021-07-10T09:42:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.status 2021-07-10T09:42:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2021-07-10T09:42:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=215 2021-07-10T09:42:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.status 2021-07-10T09:42:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.status 2021-07-10T09:42:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2021-07-10T09:42:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 215 = '' 2021-07-10T09:42:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.status 2021-07-10T09:42:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.status 2021-07-10T09:42:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2021-07-10T09:42:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 215 = '' 2021-07-10T09:42:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.status 2021-07-10T09:42:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.status 2021-07-10T09:42:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2021-07-10T09:42:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=215 2021-07-10T09:42:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 215 = 215 2021-07-10T09:42:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2021-07-10T09:42:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2021-07-10T09:42:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_33': 2021-07-10T09:42:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a41d655e46 2021-07-10T09:42:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a41d655e46 2021-07-10T09:42:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:36:10 2021 2021-07-10T09:42:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:36:10 2021 2021-07-10T09:42:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:36:10 2021 2021-07-10T09:42:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: b421f5e6-de31-4b1f-934c-397a5c2c5012 2021-07-10T09:42:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_33 snap 2021-07-10T09:42:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:42:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.export 2021-07-10T09:42:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.export 2021-07-10T09:42:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.export 2021-07-10T09:42:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.export 2021-07-10T09:42:42.592 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:42:42.591+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:42:42.594 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:42:42.593+0000 7f69e7730700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:42:42.594 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:42:42.593+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 7 2021-07-10T09:42:42.594 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:42:42.593+0000 7f69e9734700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 7 2021-07-10T09:42:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.export 2021-07-10T09:42:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.export 2021-07-10T09:42:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_33.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_33.export 2021-07-10T09:42:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_34 2021-07-10T09:42:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_34 snap 2021-07-10T09:42:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2021-07-10T09:42:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2021-07-10T09:42:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2021-07-10T09:42:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2021-07-10T09:42:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2021-07-10T09:42:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2021-07-10T09:42:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2021-07-10T09:42:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:42:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:42:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2021-07-10T09:42:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:42:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:42:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:42:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:42:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:42:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:42:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:42:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:42:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:42:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:42:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:42:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:42:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:42:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2021-07-10T09:42:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2021-07-10T09:42:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2021-07-10T09:42:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_34.status 2021-07-10T09:42:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_34.status 2021-07-10T09:42:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.status 2021-07-10T09:42:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_34.status 2021-07-10T09:42:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_34.status 2021-07-10T09:42:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2021-07-10T09:42:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2021-07-10T09:42:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 218 2021-07-10T09:42:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.status 2021-07-10T09:42:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.status 2021-07-10T09:42:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2021-07-10T09:42:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=218 2021-07-10T09:42:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2021-07-10T09:42:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 218 = '' 2021-07-10T09:42:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2021-07-10T09:42:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 218 = '' 2021-07-10T09:42:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2021-07-10T09:42:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 218 = '' 2021-07-10T09:42:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:42:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.status 2021-07-10T09:42:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2021-07-10T09:42:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=218 2021-07-10T09:42:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 218 = 218 2021-07-10T09:42:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2021-07-10T09:42:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2021-07-10T09:42:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_34': 2021-07-10T09:42:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:42:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f46cdeb41 2021-07-10T09:42:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f46cdeb41 2021-07-10T09:42:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:42:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:42:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:36:23 2021 2021-07-10T09:42:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:36:23 2021 2021-07-10T09:42:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:36:23 2021 2021-07-10T09:42:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: eea17d25-a5d2-446c-980f-6f2a02c245d4 2021-07-10T09:42:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_34 snap 2021-07-10T09:42:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:42:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.export 2021-07-10T09:42:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.export 2021-07-10T09:42:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.export 2021-07-10T09:42:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.export 2021-07-10T09:42:50.279 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:50.277+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2021-07-10T09:42:50.279 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:50.277+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2021-07-10T09:42:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.export 2021-07-10T09:42:50.428 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.425+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:42:50.428 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.425+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:42:50.431 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.429+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:42:50.452 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.450+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:42:50.452 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.451+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:42:50.452 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.451+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:42:50.453 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.451+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:42:50.453 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.452+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 9 2021-07-10T09:42:50.481 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.479+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:42:50.482 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.480+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:42:50.482 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.480+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:42:50.482 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.480+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2021-07-10T09:42:50.552 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.549+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:42:50.555 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:50.553+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:42:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.export 2021-07-10T09:42:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_34.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_34.export 2021-07-10T09:42:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_35 2021-07-10T09:42:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_35 snap 2021-07-10T09:42:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2021-07-10T09:42:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2021-07-10T09:42:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2021-07-10T09:42:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2021-07-10T09:42:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2021-07-10T09:42:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2021-07-10T09:42:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2021-07-10T09:42:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2021-07-10T09:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_35.status 2021-07-10T09:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_35.status 2021-07-10T09:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.status 2021-07-10T09:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_35.status 2021-07-10T09:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_35.status 2021-07-10T09:42:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.status 2021-07-10T09:42:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2021-07-10T09:42:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2021-07-10T09:42:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 221 2021-07-10T09:42:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.status 2021-07-10T09:42:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.status 2021-07-10T09:42:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2021-07-10T09:42:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=221 2021-07-10T09:42:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.status 2021-07-10T09:42:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.status 2021-07-10T09:42:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2021-07-10T09:42:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 221 = '' 2021-07-10T09:42:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.status 2021-07-10T09:42:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.status 2021-07-10T09:42:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2021-07-10T09:42:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 221 = '' 2021-07-10T09:42:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:42:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.status 2021-07-10T09:42:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.status 2021-07-10T09:42:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2021-07-10T09:42:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=221 2021-07-10T09:42:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 221 = 221 2021-07-10T09:42:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2021-07-10T09:42:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:42:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2021-07-10T09:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_35': 2021-07-10T09:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 3 2021-07-10T09:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d41cb5d195 2021-07-10T09:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d41cb5d195 2021-07-10T09:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, operations, non-primary 2021-07-10T09:42:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: snap-trash 2021-07-10T09:42:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:42:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:36:33 2021 2021-07-10T09:42:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:36:33 2021 2021-07-10T09:42:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:36:33 2021 2021-07-10T09:42:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:42:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:42:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:42:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: e8d3f1a9-a525-4216-a43b-9b498d357b97 2021-07-10T09:42:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:42:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_35 snap 2021-07-10T09:42:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:42:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:42:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.export 2021-07-10T09:42:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.export 2021-07-10T09:42:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.export 2021-07-10T09:42:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.export 2021-07-10T09:42:55.866 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:55.865+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:42:55.868 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:42:55.866+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:42:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.export 2021-07-10T09:42:56.055 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:42:56.054+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:42:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:42:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.export 2021-07-10T09:42:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_35.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_35.export 2021-07-10T09:42:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:42:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_36 2021-07-10T09:42:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_36 snap 2021-07-10T09:42:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:42:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:42:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2021-07-10T09:42:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2021-07-10T09:42:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:42:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:42:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:42:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2021-07-10T09:42:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:42:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:42:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:42:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2021-07-10T09:42:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:42:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:42:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:42:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:42:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:42:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:42:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:42:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:42:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:42:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:42:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2021-07-10T09:42:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:42:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:42:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:42:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:42:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:42:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:42:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:42:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:42:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:42:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:42:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:42:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:42:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:42:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:42:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:42:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:42:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:42:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:42:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:42:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:42:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:42:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:42:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2021-07-10T09:42:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:42:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:42:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:42:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:42:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:42:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:42:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:42:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:42:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2021-07-10T09:42:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2021-07-10T09:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_36.status 2021-07-10T09:42:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_36.status 2021-07-10T09:42:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.status 2021-07-10T09:42:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_36.status 2021-07-10T09:42:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_36.status 2021-07-10T09:42:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:42:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.status 2021-07-10T09:42:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2021-07-10T09:42:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2021-07-10T09:42:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 224 2021-07-10T09:42:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.status 2021-07-10T09:42:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:42:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.status 2021-07-10T09:42:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2021-07-10T09:42:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:42:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=224 2021-07-10T09:42:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:42:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.status 2021-07-10T09:42:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.status 2021-07-10T09:42:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2021-07-10T09:42:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 224 = '' 2021-07-10T09:42:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:42:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.status 2021-07-10T09:42:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:42:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:42:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:42:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.status 2021-07-10T09:42:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2021-07-10T09:42:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:42:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:42:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:42:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:42:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:42:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 224 = '' 2021-07-10T09:42:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:42:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:43:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.status 2021-07-10T09:43:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:43:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.status 2021-07-10T09:43:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2021-07-10T09:43:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=224 2021-07-10T09:43:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 224 = 224 2021-07-10T09:43:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2021-07-10T09:43:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:43:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:43:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2021-07-10T09:43:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_36': 2021-07-10T09:43:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:43:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:43:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 3 2021-07-10T09:43:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e9bf209f28 2021-07-10T09:43:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e9bf209f28 2021-07-10T09:43:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:43:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, operations, non-primary 2021-07-10T09:43:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: snap-trash 2021-07-10T09:43:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:36:45 2021 2021-07-10T09:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:36:45 2021 2021-07-10T09:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:36:45 2021 2021-07-10T09:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:43:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:43:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 8f66e46a-e345-413d-89fe-1194dc688173 2021-07-10T09:43:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:43:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_36 snap 2021-07-10T09:43:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:43:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:43:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.export 2021-07-10T09:43:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.export 2021-07-10T09:43:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.export 2021-07-10T09:43:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.export 2021-07-10T09:43:01.440 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:01.439+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:01.551 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:43:01.550+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2021-07-10T09:43:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.export 2021-07-10T09:43:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.export 2021-07-10T09:43:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_36.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_36.export 2021-07-10T09:43:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:43:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_37 2021-07-10T09:43:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_37 snap 2021-07-10T09:43:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:43:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2021-07-10T09:43:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2021-07-10T09:43:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2021-07-10T09:43:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2021-07-10T09:43:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2021-07-10T09:43:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2021-07-10T09:43:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2021-07-10T09:43:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2021-07-10T09:43:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:43:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:43:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:43:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:43:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:43:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:43:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:43:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:43:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:43:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:43:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:43:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:43:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2021-07-10T09:43:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:43:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:43:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:43:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:43:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:43:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2021-07-10T09:43:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:43:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:43:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2021-07-10T09:43:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_37.status 2021-07-10T09:43:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_37.status 2021-07-10T09:43:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.status 2021-07-10T09:43:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_37.status 2021-07-10T09:43:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_37.status 2021-07-10T09:43:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2021-07-10T09:43:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2021-07-10T09:43:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 227 2021-07-10T09:43:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.status 2021-07-10T09:43:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.status 2021-07-10T09:43:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2021-07-10T09:43:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:43:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=227 2021-07-10T09:43:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:43:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2021-07-10T09:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=104 2021-07-10T09:43:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 227 = 104 2021-07-10T09:43:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:43:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2021-07-10T09:43:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=104 2021-07-10T09:43:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 227 = 104 2021-07-10T09:43:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:43:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2021-07-10T09:43:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 227 = '' 2021-07-10T09:43:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:43:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.status 2021-07-10T09:43:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2021-07-10T09:43:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=227 2021-07-10T09:43:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 227 = 227 2021-07-10T09:43:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2021-07-10T09:43:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:43:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2021-07-10T09:43:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_37': 2021-07-10T09:43:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:43:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:43:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:43:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a42a4baa61 2021-07-10T09:43:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a42a4baa61 2021-07-10T09:43:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:43:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:43:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:43:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:43:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:36:51 2021 2021-07-10T09:43:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:36:51 2021 2021-07-10T09:43:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:36:51 2021 2021-07-10T09:43:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:43:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:43:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:43:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 6d269fc8-ff35-4029-9b78-f4587abfb513 2021-07-10T09:43:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:43:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_37 snap 2021-07-10T09:43:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:43:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:43:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.export 2021-07-10T09:43:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.export 2021-07-10T09:43:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.export 2021-07-10T09:43:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.export 2021-07-10T09:43:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.export 2021-07-10T09:43:08.643 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:43:08.642+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:43:08.647 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:43:08.644+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:43:08.798 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:43:08.797+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:43:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.export 2021-07-10T09:43:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_37.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_37.export 2021-07-10T09:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_38 2021-07-10T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_38 snap 2021-07-10T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:43:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2021-07-10T09:43:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2021-07-10T09:43:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:43:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2021-07-10T09:43:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2021-07-10T09:43:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2021-07-10T09:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2021-07-10T09:43:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2021-07-10T09:43:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2021-07-10T09:43:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:43:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:43:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:43:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:43:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:43:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:43:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:43:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:43:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:43:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:43:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:43:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:43:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2021-07-10T09:43:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2021-07-10T09:43:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2021-07-10T09:43:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:43:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:43:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:43:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:43:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:43:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:43:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:43:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:43:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:43:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:43:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:43:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:43:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2021-07-10T09:43:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:43:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:43:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:43:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:43:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:43:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2021-07-10T09:43:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:43:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:43:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2021-07-10T09:43:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_38.status 2021-07-10T09:43:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_38.status 2021-07-10T09:43:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.status 2021-07-10T09:43:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_38.status 2021-07-10T09:43:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_38.status 2021-07-10T09:43:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.status 2021-07-10T09:43:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2021-07-10T09:43:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2021-07-10T09:43:11.709 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.679133196843177) > weight (0.25) 2021-07-10T09:43:11.709 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9945008199194137) > weight (0.25) 2021-07-10T09:43:11.709 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6300988968406804) > weight (0.25) 2021-07-10T09:43:11.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7260052592570437) > weight (0.25) 2021-07-10T09:43:11.710 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2021-07-10T09:43:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 230 2021-07-10T09:43:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.status 2021-07-10T09:43:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.status 2021-07-10T09:43:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2021-07-10T09:43:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:43:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=230 2021-07-10T09:43:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:43:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.status 2021-07-10T09:43:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.status 2021-07-10T09:43:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2021-07-10T09:43:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 230 = '' 2021-07-10T09:43:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:43:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.status 2021-07-10T09:43:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.status 2021-07-10T09:43:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2021-07-10T09:43:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 230 = '' 2021-07-10T09:43:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:43:12.881 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:12.879+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.status 2021-07-10T09:43:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.status 2021-07-10T09:43:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2021-07-10T09:43:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=230 2021-07-10T09:43:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 230 = 230 2021-07-10T09:43:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2021-07-10T09:43:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:43:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2021-07-10T09:43:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_38': 2021-07-10T09:43:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:43:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:43:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:43:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f9025b77a 2021-07-10T09:43:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f9025b77a 2021-07-10T09:43:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:43:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:43:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:43:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:43:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:37:00 2021 2021-07-10T09:43:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:37:00 2021 2021-07-10T09:43:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:37:00 2021 2021-07-10T09:43:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:43:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:43:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:43:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 196bc190-401d-4754-8d72-74bea6a878ef 2021-07-10T09:43:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:43:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_38 snap 2021-07-10T09:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:43:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.export 2021-07-10T09:43:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.export 2021-07-10T09:43:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.export 2021-07-10T09:43:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.export 2021-07-10T09:43:14.493 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:43:14.491+0000 7f5115a97700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:43:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.export 2021-07-10T09:43:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.export 2021-07-10T09:43:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_38.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_38.export 2021-07-10T09:43:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:43:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_39 2021-07-10T09:43:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_39 snap 2021-07-10T09:43:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:43:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2021-07-10T09:43:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2021-07-10T09:43:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:43:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2021-07-10T09:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2021-07-10T09:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2021-07-10T09:43:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2021-07-10T09:43:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2021-07-10T09:43:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2021-07-10T09:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:43:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:43:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:43:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:43:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:43:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2021-07-10T09:43:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2021-07-10T09:43:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2021-07-10T09:43:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:43:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:43:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:43:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:43:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:43:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:43:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:43:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:43:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:43:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2021-07-10T09:43:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2021-07-10T09:43:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2021-07-10T09:43:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:43:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:43:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:43:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:43:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:43:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:43:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:43:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:43:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:43:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:43:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:43:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:43:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:43:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:43:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:43:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:43:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2021-07-10T09:43:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:43:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:43:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:43:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:43:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:43:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2021-07-10T09:43:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:43:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:43:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2021-07-10T09:43:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_39.status 2021-07-10T09:43:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_39.status 2021-07-10T09:43:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.status 2021-07-10T09:43:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_39.status 2021-07-10T09:43:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_39.status 2021-07-10T09:43:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2021-07-10T09:43:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2021-07-10T09:43:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 233 2021-07-10T09:43:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.status 2021-07-10T09:43:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.status 2021-07-10T09:43:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2021-07-10T09:43:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:43:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=233 2021-07-10T09:43:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:43:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2021-07-10T09:43:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 233 = '' 2021-07-10T09:43:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:43:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2021-07-10T09:43:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 233 = '' 2021-07-10T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:43:18.920 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6675312293001504) > weight (0.25) 2021-07-10T09:43:18.921 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7476301264917579) > weight (0.25) 2021-07-10T09:43:18.921 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2021-07-10T09:43:18.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2021-07-10T09:43:18.922 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2021-07-10T09:43:18.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:2021-07-10T09:43:18.922+0000 7f29e388a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 57637) UID: 1000 2021-07-10T09:43:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2021-07-10T09:43:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 233 = '' 2021-07-10T09:43:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:43:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.status 2021-07-10T09:43:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2021-07-10T09:43:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=233 2021-07-10T09:43:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 233 = 233 2021-07-10T09:43:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2021-07-10T09:43:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:43:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2021-07-10T09:43:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_39': 2021-07-10T09:43:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:43:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:43:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:43:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d411683800 2021-07-10T09:43:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d411683800 2021-07-10T09:43:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:43:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:43:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:43:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:43:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:37:15 2021 2021-07-10T09:43:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:37:15 2021 2021-07-10T09:43:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:37:15 2021 2021-07-10T09:43:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:43:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:43:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:43:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: c7b946a3-39c6-493e-9ac0-9b1ced62c995 2021-07-10T09:43:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:43:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_39 snap 2021-07-10T09:43:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:43:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:43:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.export 2021-07-10T09:43:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.export 2021-07-10T09:43:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.export 2021-07-10T09:43:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.export 2021-07-10T09:43:22.121 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:43:22.120+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:43:22.123 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:43:22.123+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:43:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.export 2021-07-10T09:43:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.export 2021-07-10T09:43:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_39.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_39.export 2021-07-10T09:43:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:43:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_40 2021-07-10T09:43:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_40 snap 2021-07-10T09:43:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:43:22.923 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2021-07-10T09:43:22.923 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:43:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2021-07-10T09:43:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2021-07-10T09:43:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:43:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2021-07-10T09:43:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2021-07-10T09:43:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2021-07-10T09:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2021-07-10T09:43:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:43:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:43:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:43:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:43:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:43:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2021-07-10T09:43:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2021-07-10T09:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_40.status 2021-07-10T09:43:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_40.status 2021-07-10T09:43:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.status 2021-07-10T09:43:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_40.status 2021-07-10T09:43:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_40.status 2021-07-10T09:43:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2021-07-10T09:43:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2021-07-10T09:43:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 236 2021-07-10T09:43:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.status 2021-07-10T09:43:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.status 2021-07-10T09:43:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2021-07-10T09:43:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:43:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=236 2021-07-10T09:43:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:43:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2021-07-10T09:43:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=107 2021-07-10T09:43:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 236 = 107 2021-07-10T09:43:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:43:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2021-07-10T09:43:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=107 2021-07-10T09:43:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 236 = 107 2021-07-10T09:43:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:43:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2021-07-10T09:43:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 236 = '' 2021-07-10T09:43:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:43:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.status 2021-07-10T09:43:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2021-07-10T09:43:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=236 2021-07-10T09:43:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 236 = 236 2021-07-10T09:43:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2021-07-10T09:43:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:43:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2021-07-10T09:43:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_40': 2021-07-10T09:43:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:43:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:43:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:43:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e95ac190f8 2021-07-10T09:43:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e95ac190f8 2021-07-10T09:43:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:43:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:43:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:43:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:43:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:37:29 2021 2021-07-10T09:43:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:37:29 2021 2021-07-10T09:43:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:37:29 2021 2021-07-10T09:43:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:43:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:43:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:43:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 95ff3995-ee74-466c-80cd-fa9a3b8bbf34 2021-07-10T09:43:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:43:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_40 snap 2021-07-10T09:43:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:43:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:43:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.export 2021-07-10T09:43:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.export 2021-07-10T09:43:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.export 2021-07-10T09:43:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.export 2021-07-10T09:43:29.492 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:29.491+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:29.538 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:29.538+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.export 2021-07-10T09:43:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.export 2021-07-10T09:43:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_40.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_40.export 2021-07-10T09:43:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:43:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_41 2021-07-10T09:43:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_41 snap 2021-07-10T09:43:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:43:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2021-07-10T09:43:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2021-07-10T09:43:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:43:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:30.858 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:43:30.857+0000 7f4236192700 -1 reset not still connected to 0x55a45b5831e0 2021-07-10T09:43:30.859 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:43:30.858+0000 7f4236192700 -1 reset not still connected to 0x55a45b815790 2021-07-10T09:43:30.859 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:43:30.858+0000 7f4236192700 -1 reset not still connected to 0x55a45b815e10 2021-07-10T09:43:30.860 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:43:30.858+0000 7f4236192700 -1 reset not still connected to 0x55a45ebf0820 2021-07-10T09:43:30.860 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:43:30.858+0000 7f4236192700 -1 reset not still connected to 0x55a46f37fad0 2021-07-10T09:43:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2021-07-10T09:43:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2021-07-10T09:43:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2021-07-10T09:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2021-07-10T09:43:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2021-07-10T09:43:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2021-07-10T09:43:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:43:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:43:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:43:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:43:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:43:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:43:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:43:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:43:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:43:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:43:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:43:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:43:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2021-07-10T09:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:43:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:43:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2021-07-10T09:43:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:43:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:43:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2021-07-10T09:43:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_41.status 2021-07-10T09:43:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_41.status 2021-07-10T09:43:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.status 2021-07-10T09:43:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_41.status 2021-07-10T09:43:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_41.status 2021-07-10T09:43:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2021-07-10T09:43:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2021-07-10T09:43:32.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 239 2021-07-10T09:43:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.status 2021-07-10T09:43:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.status 2021-07-10T09:43:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2021-07-10T09:43:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:43:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=239 2021-07-10T09:43:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:43:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2021-07-10T09:43:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=108 2021-07-10T09:43:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 239 = 108 2021-07-10T09:43:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2021-07-10T09:43:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 239 = '' 2021-07-10T09:43:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:43:34.049 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:34.047+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2021-07-10T09:43:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:34.229 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:34.228+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:34.230 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:34.228+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 239 = '' 2021-07-10T09:43:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:43:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.status 2021-07-10T09:43:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2021-07-10T09:43:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=239 2021-07-10T09:43:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 239 = 239 2021-07-10T09:43:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2021-07-10T09:43:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:43:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2021-07-10T09:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_41': 2021-07-10T09:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:43:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a446a2947e 2021-07-10T09:43:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a446a2947e 2021-07-10T09:43:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:43:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:43:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:43:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:43:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:37:41 2021 2021-07-10T09:43:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:37:41 2021 2021-07-10T09:43:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:37:41 2021 2021-07-10T09:43:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:43:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:43:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:43:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 4b2d81b9-8236-424c-b07d-de4ead20c712 2021-07-10T09:43:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:43:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_41 snap 2021-07-10T09:43:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:43:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:43:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.export 2021-07-10T09:43:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.export 2021-07-10T09:43:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.export 2021-07-10T09:43:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.export 2021-07-10T09:43:37.073 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:37.071+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:37.136 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:37.134+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:43:37.136 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:37.135+0000 7f2571cc1700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.export 2021-07-10T09:43:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.export 2021-07-10T09:43:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_41.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_41.export 2021-07-10T09:43:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:43:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_42 2021-07-10T09:43:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_42 snap 2021-07-10T09:43:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:43:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2021-07-10T09:43:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2021-07-10T09:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2021-07-10T09:43:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2021-07-10T09:43:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2021-07-10T09:43:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2021-07-10T09:43:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2021-07-10T09:43:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2021-07-10T09:43:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:43:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:43:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:43:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:43:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:43:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:43:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:43:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:43:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:43:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:43:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:43:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:43:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2021-07-10T09:43:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2021-07-10T09:43:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2021-07-10T09:43:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:43:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:43:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:43:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:43:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:43:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:43:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:43:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:43:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:43:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:43:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:43:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2021-07-10T09:43:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Stopped"' 2021-07-10T09:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Stopped"' 2021-07-10T09:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = stopped 2021-07-10T09:43:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:43:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2021-07-10T09:43:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2021-07-10T09:43:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2021-07-10T09:43:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2021-07-10T09:43:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2021-07-10T09:43:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2021-07-10T09:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:43:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:43:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:43:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:43:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2021-07-10T09:43:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2021-07-10T09:43:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2021-07-10T09:43:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:43:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:43:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:43:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:43:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:43:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:43:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:43:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:43:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:43:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:43:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:43:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:43:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2021-07-10T09:43:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Stopped"' 2021-07-10T09:43:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Stopped"' 2021-07-10T09:43:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = stopped 2021-07-10T09:43:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:43:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2021-07-10T09:43:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2021-07-10T09:43:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2021-07-10T09:43:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2021-07-10T09:43:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2021-07-10T09:43:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2021-07-10T09:43:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:43:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:43:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:43:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:43:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:43:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:43:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:43:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:43:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:43:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:43:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:43:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:43:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:43:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:43:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:43:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2021-07-10T09:43:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2021-07-10T09:43:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:43:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:43:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2021-07-10T09:43:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:43:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:43:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:43:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:43:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:43:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:43:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:43:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:43:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:43:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:43:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:43:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:43:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:43:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:43:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:43:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2021-07-10T09:43:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Stopped"' 2021-07-10T09:43:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Stopped"' 2021-07-10T09:43:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = stopped 2021-07-10T09:43:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:43:48.358 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:43:48.355+0000 7f69f674e700 -1 reset not still connected to 0x5630cbbc4750 2021-07-10T09:43:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2021-07-10T09:43:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:43:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:43:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:43:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2021-07-10T09:43:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:43:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:43:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:43:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:43:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:43:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:43:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:43:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:43:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:43:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2021-07-10T09:43:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:43:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:43:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:43:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:43:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:43:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:43:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:43:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:43:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:43:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:43:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:43:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:43:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:43:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:43:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:43:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:43:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:43:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:43:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2021-07-10T09:43:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:43:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:43:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:43:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:43:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:43:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:43:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:43:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2021-07-10T09:43:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:43:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:43:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2021-07-10T09:43:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:43:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_42.status 2021-07-10T09:43:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_42.status 2021-07-10T09:43:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.status 2021-07-10T09:43:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_42.status 2021-07-10T09:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_42.status 2021-07-10T09:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2021-07-10T09:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2021-07-10T09:43:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 242 2021-07-10T09:43:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.status 2021-07-10T09:43:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.status 2021-07-10T09:43:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2021-07-10T09:43:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:43:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=242 2021-07-10T09:43:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:43:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2021-07-10T09:43:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 242 = '' 2021-07-10T09:43:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:43:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2021-07-10T09:43:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 242 = '' 2021-07-10T09:43:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:43:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2021-07-10T09:43:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:43:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:43:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:43:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:43:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:43:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 242 = '' 2021-07-10T09:43:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:43:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:43:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.status 2021-07-10T09:43:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2021-07-10T09:43:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:43:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:43:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=242 2021-07-10T09:43:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 242 = 242 2021-07-10T09:43:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2021-07-10T09:43:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:43:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:43:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2021-07-10T09:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_42': 2021-07-10T09:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f9290655d 2021-07-10T09:43:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f9290655d 2021-07-10T09:43:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:43:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:43:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:37:47 2021 2021-07-10T09:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:37:47 2021 2021-07-10T09:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:37:47 2021 2021-07-10T09:43:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:43:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:43:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:43:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 5d275fe5-f2c3-4166-9a6b-f93e82f23dbe 2021-07-10T09:43:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:43:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_42 snap 2021-07-10T09:43:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:43:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:43:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.export 2021-07-10T09:43:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.export 2021-07-10T09:43:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.export 2021-07-10T09:43:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.export 2021-07-10T09:43:59.226 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:59.221+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:59.226 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:59.222+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:43:59.226 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:59.222+0000 7f2577ccd700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:43:59.227 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:59.222+0000 7f25754c8700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:43:59.227 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:43:59.225+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:43:59.282 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:43:59.280+0000 7f69eb738700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:43:59.283 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:43:59.281+0000 7f69e6f2f700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2021-07-10T09:43:59.283 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:43:59.282+0000 7f69e7730700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2021-07-10T09:43:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.export 2021-07-10T09:43:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:43:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.export 2021-07-10T09:43:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_42.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_42.export 2021-07-10T09:43:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_43 2021-07-10T09:43:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_43 snap 2021-07-10T09:43:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:43:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:43:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:43:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:43:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2021-07-10T09:43:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:43:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2021-07-10T09:43:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:43:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:43:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:43:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:43:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:43:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:43:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2021-07-10T09:44:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2021-07-10T09:44:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2021-07-10T09:44:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2021-07-10T09:44:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2021-07-10T09:44:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2021-07-10T09:44:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:44:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:44:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:44:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:44:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:44:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:44:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:44:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:44:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:44:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:44:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:44:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:44:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2021-07-10T09:44:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2021-07-10T09:44:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2021-07-10T09:44:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_43.status 2021-07-10T09:44:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_43.status 2021-07-10T09:44:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.status 2021-07-10T09:44:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_43.status 2021-07-10T09:44:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_43.status 2021-07-10T09:44:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.status 2021-07-10T09:44:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2021-07-10T09:44:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2021-07-10T09:44:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 245 2021-07-10T09:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.status 2021-07-10T09:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.status 2021-07-10T09:44:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2021-07-10T09:44:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=245 2021-07-10T09:44:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.status 2021-07-10T09:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.status 2021-07-10T09:44:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2021-07-10T09:44:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 245 = '' 2021-07-10T09:44:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.status 2021-07-10T09:44:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.status 2021-07-10T09:44:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2021-07-10T09:44:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 245 = '' 2021-07-10T09:44:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.status 2021-07-10T09:44:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.status 2021-07-10T09:44:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2021-07-10T09:44:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=245 2021-07-10T09:44:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 245 = 245 2021-07-10T09:44:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2021-07-10T09:44:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2021-07-10T09:44:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_43': 2021-07-10T09:44:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12d46c5f748e 2021-07-10T09:44:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12d46c5f748e 2021-07-10T09:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:37:59 2021 2021-07-10T09:44:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:37:59 2021 2021-07-10T09:44:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:37:59 2021 2021-07-10T09:44:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 7ac5ff99-64a3-4213-92d3-fcfa30b216a4 2021-07-10T09:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_43 snap 2021-07-10T09:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.export 2021-07-10T09:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.export 2021-07-10T09:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.export 2021-07-10T09:44:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.export 2021-07-10T09:44:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.export 2021-07-10T09:44:04.766 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:04.763+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:44:04.766 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:04.764+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2021-07-10T09:44:04.767 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:04.764+0000 7f317305e700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:44:04.892 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:04.890+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:04.892 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:04.890+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:04.896 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:04.894+0000 7f317a86d700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:44:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.export 2021-07-10T09:44:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_43.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_43.export 2021-07-10T09:44:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_44 2021-07-10T09:44:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_44 snap 2021-07-10T09:44:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:44:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2021-07-10T09:44:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2021-07-10T09:44:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:44:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2021-07-10T09:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2021-07-10T09:44:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2021-07-10T09:44:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2021-07-10T09:44:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2021-07-10T09:44:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2021-07-10T09:44:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_44.status 2021-07-10T09:44:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_44.status 2021-07-10T09:44:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.status 2021-07-10T09:44:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_44.status 2021-07-10T09:44:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_44.status 2021-07-10T09:44:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2021-07-10T09:44:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2021-07-10T09:44:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 248 2021-07-10T09:44:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.status 2021-07-10T09:44:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.status 2021-07-10T09:44:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2021-07-10T09:44:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=248 2021-07-10T09:44:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2021-07-10T09:44:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 248 = '' 2021-07-10T09:44:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2021-07-10T09:44:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 248 = '' 2021-07-10T09:44:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:08.980 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:08.978+0000 7f2572cc3700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:08.980 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:08.978+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:08.986 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:08.984+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2021-07-10T09:44:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 248 = '' 2021-07-10T09:44:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:44:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.status 2021-07-10T09:44:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2021-07-10T09:44:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=248 2021-07-10T09:44:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 248 = 248 2021-07-10T09:44:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2021-07-10T09:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2021-07-10T09:44:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_44': 2021-07-10T09:44:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12e9f7716402 2021-07-10T09:44:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12e9f7716402 2021-07-10T09:44:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:38:06 2021 2021-07-10T09:44:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:38:06 2021 2021-07-10T09:44:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:38:06 2021 2021-07-10T09:44:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 2e6f95e6-0321-4cc6-b82c-1bf30c1ece13 2021-07-10T09:44:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_44 snap 2021-07-10T09:44:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:44:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.export 2021-07-10T09:44:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.export 2021-07-10T09:44:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.export 2021-07-10T09:44:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.export 2021-07-10T09:44:12.069 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:12.068+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:12.070 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:12.069+0000 7f25734c4700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:12.188 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:44:12.188+0000 7f69e3f29700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:44:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.export 2021-07-10T09:44:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.export 2021-07-10T09:44:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_44.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_44.export 2021-07-10T09:44:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_45 2021-07-10T09:44:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_45 snap 2021-07-10T09:44:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:44:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2021-07-10T09:44:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2021-07-10T09:44:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:44:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2021-07-10T09:44:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2021-07-10T09:44:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2021-07-10T09:44:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2021-07-10T09:44:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2021-07-10T09:44:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2021-07-10T09:44:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:44:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:44:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:44:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:44:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:44:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:44:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:44:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:44:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:44:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:44:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:44:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:44:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2021-07-10T09:44:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2021-07-10T09:44:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2021-07-10T09:44:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_45.status 2021-07-10T09:44:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_45.status 2021-07-10T09:44:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.status 2021-07-10T09:44:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_45.status 2021-07-10T09:44:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_45.status 2021-07-10T09:44:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2021-07-10T09:44:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2021-07-10T09:44:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 251 2021-07-10T09:44:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.status 2021-07-10T09:44:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.status 2021-07-10T09:44:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2021-07-10T09:44:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=251 2021-07-10T09:44:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2021-07-10T09:44:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=112 2021-07-10T09:44:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 251 = 112 2021-07-10T09:44:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2021-07-10T09:44:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=112 2021-07-10T09:44:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 251 = 112 2021-07-10T09:44:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2021-07-10T09:44:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 251 = '' 2021-07-10T09:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:44:16.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8444172014672736) > weight (0.25) 2021-07-10T09:44:16.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8448752555974379) > weight (0.25) 2021-07-10T09:44:16.712 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2935221302863146) > weight (0.25) 2021-07-10T09:44:16.712 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2021-07-10T09:44:16.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2021-07-10T09:44:16.715 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2021-07-10T09:44:16.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:2021-07-10T09:44:16.714+0000 7f32d70b9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 50624) UID: 1000 2021-07-10T09:44:16.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:44:17.010 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:17.006+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:44:17.011 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:17.006+0000 7f25784ce700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.status 2021-07-10T09:44:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2021-07-10T09:44:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=251 2021-07-10T09:44:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 251 = 251 2021-07-10T09:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2021-07-10T09:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2021-07-10T09:44:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_45': 2021-07-10T09:44:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a414af3c2b 2021-07-10T09:44:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a414af3c2b 2021-07-10T09:44:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:38:12 2021 2021-07-10T09:44:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:38:12 2021 2021-07-10T09:44:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:38:12 2021 2021-07-10T09:44:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 20c9cfeb-2031-4022-b5be-4a842bcbc84c 2021-07-10T09:44:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_45 snap 2021-07-10T09:44:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:44:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.export 2021-07-10T09:44:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.export 2021-07-10T09:44:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.export 2021-07-10T09:44:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.export 2021-07-10T09:44:19.421 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:19.420+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.export 2021-07-10T09:44:19.922 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:19.920+0000 7f3177066700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.export 2021-07-10T09:44:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_45.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_45.export 2021-07-10T09:44:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_46 2021-07-10T09:44:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_46 snap 2021-07-10T09:44:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:44:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2021-07-10T09:44:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2021-07-10T09:44:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:44:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2021-07-10T09:44:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2021-07-10T09:44:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2021-07-10T09:44:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2021-07-10T09:44:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2021-07-10T09:44:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2021-07-10T09:44:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:44:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:44:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:44:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:44:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:44:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:44:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:44:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:44:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:44:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:44:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:44:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:44:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2021-07-10T09:44:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2021-07-10T09:44:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2021-07-10T09:44:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_46.status 2021-07-10T09:44:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_46.status 2021-07-10T09:44:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.status 2021-07-10T09:44:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_46.status 2021-07-10T09:44:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_46.status 2021-07-10T09:44:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2021-07-10T09:44:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2021-07-10T09:44:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 254 2021-07-10T09:44:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.status 2021-07-10T09:44:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.status 2021-07-10T09:44:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2021-07-10T09:44:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=254 2021-07-10T09:44:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2021-07-10T09:44:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=113 2021-07-10T09:44:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 254 = 113 2021-07-10T09:44:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2021-07-10T09:44:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 254 = '' 2021-07-10T09:44:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2021-07-10T09:44:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 254 = '' 2021-07-10T09:44:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:44:25.716 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2021-07-10T09:44:25.717 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:44:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:44:25.717 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2021-07-10T09:44:25.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2021-07-10T09:44:25.718 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2021-07-10T09:44:25.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2021-07-10T09:44:25.722 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2021-07-10T09:44:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.status 2021-07-10T09:44:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2021-07-10T09:44:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=254 2021-07-10T09:44:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 254 = 254 2021-07-10T09:44:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2021-07-10T09:44:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2021-07-10T09:44:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_46': 2021-07-10T09:44:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f57d42dd7 2021-07-10T09:44:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f57d42dd7 2021-07-10T09:44:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:38:18 2021 2021-07-10T09:44:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:38:18 2021 2021-07-10T09:44:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:38:18 2021 2021-07-10T09:44:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: d8523da8-fead-4b73-9128-acc11fcefb83 2021-07-10T09:44:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_46 snap 2021-07-10T09:44:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:44:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.export 2021-07-10T09:44:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.export 2021-07-10T09:44:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.export 2021-07-10T09:44:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.export 2021-07-10T09:44:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.export 2021-07-10T09:44:28.031 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:28.029+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.export 2021-07-10T09:44:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_46.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_46.export 2021-07-10T09:44:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_47 2021-07-10T09:44:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_47 snap 2021-07-10T09:44:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:44:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2021-07-10T09:44:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2021-07-10T09:44:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:44:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2021-07-10T09:44:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2021-07-10T09:44:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2021-07-10T09:44:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2021-07-10T09:44:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2021-07-10T09:44:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2021-07-10T09:44:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:44:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:44:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:44:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:44:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:44:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:44:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:44:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:44:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:44:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:44:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:44:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:44:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2021-07-10T09:44:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2021-07-10T09:44:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2021-07-10T09:44:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:44:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:44:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:44:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:44:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:44:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:44:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:44:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:44:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:44:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:44:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:44:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:44:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:44:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:44:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:44:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:44:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:44:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:44:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:44:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2021-07-10T09:44:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 10 closest matches: 2021-07-10T09:44:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_10 2021-07-10T09:44:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_11 2021-07-10T09:44:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_17 2021-07-10T09:44:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_18 2021-07-10T09:44:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_22 2021-07-10T09:44:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_23 2021-07-10T09:44:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_29 2021-07-10T09:44:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_30 2021-07-10T09:44:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_31 2021-07-10T09:44:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_38 2021-07-10T09:44:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2021-07-10T09:44:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:44:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:44:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:44:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:44:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:44:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:44:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:44:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:44:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:44:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:44:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:44:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:44:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:44:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:44:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:44:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:44:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:44:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:44:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:44:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2021-07-10T09:44:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2021-07-10T09:44:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2021-07-10T09:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_47.status 2021-07-10T09:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_47.status 2021-07-10T09:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.status 2021-07-10T09:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_47.status 2021-07-10T09:44:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_47.status 2021-07-10T09:44:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2021-07-10T09:44:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2021-07-10T09:44:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 257 2021-07-10T09:44:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.status 2021-07-10T09:44:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.status 2021-07-10T09:44:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2021-07-10T09:44:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=257 2021-07-10T09:44:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2021-07-10T09:44:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=114 2021-07-10T09:44:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 257 = 114 2021-07-10T09:44:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2021-07-10T09:44:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=114 2021-07-10T09:44:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 257 = 114 2021-07-10T09:44:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2021-07-10T09:44:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:32.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 257 = '' 2021-07-10T09:44:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:44:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.status 2021-07-10T09:44:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2021-07-10T09:44:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=257 2021-07-10T09:44:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 257 = 257 2021-07-10T09:44:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2021-07-10T09:44:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2021-07-10T09:44:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_47': 2021-07-10T09:44:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12fd527d2ecc 2021-07-10T09:44:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12fd527d2ecc 2021-07-10T09:44:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:38:29 2021 2021-07-10T09:44:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:38:29 2021 2021-07-10T09:44:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:38:29 2021 2021-07-10T09:44:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 15151574-c70a-4526-ac8d-20af3978894c 2021-07-10T09:44:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_47 snap 2021-07-10T09:44:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:44:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.export 2021-07-10T09:44:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.export 2021-07-10T09:44:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.export 2021-07-10T09:44:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.export 2021-07-10T09:44:35.505 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:44:35.503+0000 7f5114a95700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2021-07-10T09:44:35.506 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:44:35.504+0000 7f5111a8f700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:44:35.568 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:35.562+0000 7f2573cc5700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:44:35.568 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:44:35.562+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:44:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.export 2021-07-10T09:44:35.670 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:35.669+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:35.775 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:35.775+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:44:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.export 2021-07-10T09:44:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_47.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_47.export 2021-07-10T09:44:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_48 2021-07-10T09:44:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_48 snap 2021-07-10T09:44:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:44:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2021-07-10T09:44:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2021-07-10T09:44:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:44:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2021-07-10T09:44:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2021-07-10T09:44:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2021-07-10T09:44:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2021-07-10T09:44:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2021-07-10T09:44:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2021-07-10T09:44:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:44:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:44:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:44:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:44:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:44:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:44:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:44:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:44:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:44:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:44:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2021-07-10T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2021-07-10T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2021-07-10T09:44:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:2 2021-07-10T09:44:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:2 cluster instance 2021-07-10T09:44:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:2 2021-07-10T09:44:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=2 2021-07-10T09:44:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 2 = cluster1:2 ']' 2021-07-10T09:44:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=2 2021-07-10T09:44:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:44:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:2 cluster1 2021-07-10T09:44:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:2 2021-07-10T09:44:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2021-07-10T09:44:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:2 2021-07-10T09:44:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=2 2021-07-10T09:44:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 2 = cluster1:2 ']' 2021-07-10T09:44:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=2 2021-07-10T09:44:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=2 2021-07-10T09:44:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2021-07-10T09:44:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:44:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:44:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-07-10T09:44:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2021-07-10T09:44:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:no valid command found; 10 closest matches: 2021-07-10T09:44:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_10 2021-07-10T09:44:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_11 2021-07-10T09:44:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_17 2021-07-10T09:44:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_18 2021-07-10T09:44:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_22 2021-07-10T09:44:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_23 2021-07-10T09:44:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_29 2021-07-10T09:44:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_30 2021-07-10T09:44:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_31 2021-07-10T09:44:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status mirror/image_38 2021-07-10T09:44:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2021-07-10T09:44:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:3 2021-07-10T09:44:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:3 cluster instance 2021-07-10T09:44:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:3 2021-07-10T09:44:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=3 2021-07-10T09:44:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 3 = cluster1:3 ']' 2021-07-10T09:44:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=3 2021-07-10T09:44:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:44:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:3 cluster1 2021-07-10T09:44:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:3 2021-07-10T09:44:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2021-07-10T09:44:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:3 2021-07-10T09:44:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=3 2021-07-10T09:44:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 3 = cluster1:3 ']' 2021-07-10T09:44:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=3 2021-07-10T09:44:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=3 2021-07-10T09:44:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2021-07-10T09:44:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:44:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:44:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-07-10T09:44:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2021-07-10T09:44:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2021-07-10T09:44:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2021-07-10T09:44:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_48.status 2021-07-10T09:44:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_48.status 2021-07-10T09:44:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.status 2021-07-10T09:44:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_48.status 2021-07-10T09:44:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_48.status 2021-07-10T09:44:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2021-07-10T09:44:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2021-07-10T09:44:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 260 2021-07-10T09:44:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.status 2021-07-10T09:44:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.status 2021-07-10T09:44:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2021-07-10T09:44:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=260 2021-07-10T09:44:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2021-07-10T09:44:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=115 2021-07-10T09:44:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 260 = 115 2021-07-10T09:44:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2021-07-10T09:44:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=115 2021-07-10T09:44:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 260 = 115 2021-07-10T09:44:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2021-07-10T09:44:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 260 = '' 2021-07-10T09:44:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:44:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.status 2021-07-10T09:44:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2021-07-10T09:44:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=260 2021-07-10T09:44:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 260 = 260 2021-07-10T09:44:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2021-07-10T09:44:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2021-07-10T09:44:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_48': 2021-07-10T09:44:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 12fd8c41072e 2021-07-10T09:44:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.12fd8c41072e 2021-07-10T09:44:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:38:44 2021 2021-07-10T09:44:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:38:44 2021 2021-07-10T09:44:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:38:44 2021 2021-07-10T09:44:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: f72a72bf-cdbb-4c85-9a57-4fc62c0316f4 2021-07-10T09:44:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_48 snap 2021-07-10T09:44:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:44:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.export 2021-07-10T09:44:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.export 2021-07-10T09:44:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.export 2021-07-10T09:44:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.export 2021-07-10T09:44:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.export 2021-07-10T09:44:42.692 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.691+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:42.693 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.691+0000 7f3176064700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:42.693 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.692+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:42.693 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.692+0000 7f317385f700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:44:42.719 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.717+0000 7f3174861700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:42.719 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.717+0000 7f317906a700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:42.720 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.718+0000 7f3175062700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:44:42.720 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.718+0000 7f317a06c700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2021-07-10T09:44:42.721 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.719+0000 7f3174060700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2021-07-10T09:44:42.721 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.720+0000 7f3177867700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2021-07-10T09:44:42.722 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.721+0000 7f3178869700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2021-07-10T09:44:42.807 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:44:42.803+0000 7f3178068700 -1 bdev(0x55911fbaa000 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2021-07-10T09:44:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.export 2021-07-10T09:44:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_48.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_48.export 2021-07-10T09:44:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_49 2021-07-10T09:44:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_49 snap 2021-07-10T09:44:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:44:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2021-07-10T09:44:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2021-07-10T09:44:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:44:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2021-07-10T09:44:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2021-07-10T09:44:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2021-07-10T09:44:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2021-07-10T09:44:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2021-07-10T09:44:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd mirror status : get status for rbd mirror 2021-07-10T09:44:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:admin_socket: invalid command 2021-07-10T09:44:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2021-07-10T09:44:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:1 2021-07-10T09:44:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:1 cluster instance 2021-07-10T09:44:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:1 2021-07-10T09:44:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=1 2021-07-10T09:44:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=1 2021-07-10T09:44:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:1 cluster1 2021-07-10T09:44:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:1 2021-07-10T09:44:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2021-07-10T09:44:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:1 2021-07-10T09:44:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=1 2021-07-10T09:44:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 1 = cluster1:1 ']' 2021-07-10T09:44:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=1 2021-07-10T09:44:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=1 2021-07-10T09:44:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2021-07-10T09:44:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-07-10T09:44:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2021-07-10T09:44:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2021-07-10T09:44:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2021-07-10T09:44:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_49.status 2021-07-10T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_49.status 2021-07-10T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.status 2021-07-10T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_49.status 2021-07-10T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_49.status 2021-07-10T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2021-07-10T09:44:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2021-07-10T09:44:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 263 2021-07-10T09:44:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.status 2021-07-10T09:44:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.status 2021-07-10T09:44:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2021-07-10T09:44:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=263 2021-07-10T09:44:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2021-07-10T09:44:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=116 2021-07-10T09:44:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 263 = 116 2021-07-10T09:44:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2021-07-10T09:44:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=116 2021-07-10T09:44:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 263 = 116 2021-07-10T09:44:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2021-07-10T09:44:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 263 = '' 2021-07-10T09:44:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:44:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.status 2021-07-10T09:44:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2021-07-10T09:44:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=263 2021-07-10T09:44:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 263 = 263 2021-07-10T09:44:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2021-07-10T09:44:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2021-07-10T09:44:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_49': 2021-07-10T09:44:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 11a46ad5d26c 2021-07-10T09:44:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.11a46ad5d26c 2021-07-10T09:44:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:38:56 2021 2021-07-10T09:44:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:38:56 2021 2021-07-10T09:44:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:38:56 2021 2021-07-10T09:44:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: 498b05d2-36de-42fa-a5cc-a812fdb57d2a 2021-07-10T09:44:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_49 snap 2021-07-10T09:44:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:44:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.export 2021-07-10T09:44:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.export 2021-07-10T09:44:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.export 2021-07-10T09:44:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.export 2021-07-10T09:44:49.469 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:44:49.468+0000 7f69e7730700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:44:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.export 2021-07-10T09:44:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.export 2021-07-10T09:44:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_49.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_49.export 2021-07-10T09:44:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_50 2021-07-10T09:44:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_snap cluster2 mirror image_50 snap 2021-07-10T09:44:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2021-07-10T09:44:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2021-07-10T09:44:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2021-07-10T09:44:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=started 2021-07-10T09:44:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2021-07-10T09:44:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=started 2021-07-10T09:44:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_result 2021-07-10T09:44:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=stopped 2021-07-10T09:44:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2021-07-10T09:44:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep -i state 2021-07-10T09:44:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_instance=cluster1 2021-07-10T09:44:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:44:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance=cluster1 2021-07-10T09:44:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local loop_instance 2021-07-10T09:44:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2021-07-10T09:44:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sleep 0 2021-07-10T09:44:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' cluster1 '!=' cluster1 ']' 2021-07-10T09:44:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ seq 0 3 2021-07-10T09:44:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2021-07-10T09:44:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2021-07-10T09:44:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1:0 2021-07-10T09:44:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ shift 2021-07-10T09:44:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance 2021-07-10T09:44:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set_cluster_instance cluster1:0 cluster instance 2021-07-10T09:44:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local val=cluster1:0 2021-07-10T09:44:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster_var_name=cluster 2021-07-10T09:44:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local instance_var_name=instance 2021-07-10T09:44:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:44:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:44:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval cluster=cluster1 2021-07-10T09:44:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ eval instance=0 2021-07-10T09:44:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ daemon_asok_file cluster1:0 cluster1 2021-07-10T09:44:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local local_cluster=cluster1:0 2021-07-10T09:44:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster=cluster1 2021-07-10T09:44:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance 2021-07-10T09:44:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2021-07-10T09:44:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local val=cluster1:0 2021-07-10T09:44:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local cluster_var_name=local_cluster 2021-07-10T09:44:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ local instance_var_name=instance 2021-07-10T09:44:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ cluster=cluster1 2021-07-10T09:44:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ instance=0 2021-07-10T09:44:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ '[' 0 = cluster1:0 ']' 2021-07-10T09:44:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval local_cluster=cluster1 2021-07-10T09:44:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ local_cluster=cluster1 2021-07-10T09:44:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ eval instance=0 2021-07-10T09:44:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ instance=0 2021-07-10T09:44:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2021-07-10T09:44:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-07-10T09:44:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2021-07-10T09:44:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ return 0 2021-07-10T09:44:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ status_result=' "state": "Replaying"' 2021-07-10T09:44:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo ' "state": "Replaying"' 2021-07-10T09:44:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ grep -i Replaying 2021-07-10T09:44:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: "state": "Replaying" 2021-07-10T09:44:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=started 2021-07-10T09:44:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test started = started 2021-07-10T09:44:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2021-07-10T09:44:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = journal ']' 2021-07-10T09:44:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' snapshot = snapshot ']' 2021-07-10T09:44:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2021-07-10T09:44:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_cluster=cluster1 2021-07-10T09:44:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster2-mirror-image_50.status 2021-07-10T09:44:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster2-mirror-image_50.status 2021-07-10T09:44:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local status_log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.status 2021-07-10T09:44:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ mkfname cluster1-mirror-image_50.status 2021-07-10T09:44:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo cluster1-mirror-image_50.status 2021-07-10T09:44:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -e 's|[/ ]|_|g' 2021-07-10T09:44:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local local_status_log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2021-07-10T09:44:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2021-07-10T09:44:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Snapshot ID: 266 2021-07-10T09:44:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.status 2021-07-10T09:44:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.status 2021-07-10T09:44:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2021-07-10T09:44:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/id 2021-07-10T09:44:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snapshot_id=266 2021-07-10T09:44:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.2 2021-07-10T09:44:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2021-07-10T09:44:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=117 2021-07-10T09:44:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 266 = 117 2021-07-10T09:44:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.4 2021-07-10T09:44:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2021-07-10T09:44:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=117 2021-07-10T09:44:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 266 = 117 2021-07-10T09:44:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.8 2021-07-10T09:44:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2021-07-10T09:44:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:44:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:-:1.1: Document is empty 2021-07-10T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: 2021-07-10T09:44:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:^ 2021-07-10T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id= 2021-07-10T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 266 = '' 2021-07-10T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2021-07-10T09:44:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1.6 2021-07-10T09:44:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local log=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.status 2021-07-10T09:44:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2021-07-10T09:44:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2021-07-10T09:44:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2021-07-10T09:44:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local primary_snapshot_id=266 2021-07-10T09:44:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 266 = 266 2021-07-10T09:44:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2021-07-10T09:44:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:44:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:44:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2021-07-10T09:44:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:44:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2021-07-10T09:44:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd image 'image_50': 2021-07-10T09:44:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: size 128 MiB in 32 objects 2021-07-10T09:44:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: order 22 (4 MiB objects) 2021-07-10T09:44:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: snapshot_count: 2 2021-07-10T09:44:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: id: 122f9bcb7c50 2021-07-10T09:44:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: block_name_prefix: rbd_data.122f9bcb7c50 2021-07-10T09:44:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: format: 2 2021-07-10T09:44:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: features: layering, non-primary 2021-07-10T09:44:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: op_features: 2021-07-10T09:44:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: flags: 2021-07-10T09:44:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: create_timestamp: Sat Jul 10 09:39:05 2021 2021-07-10T09:44:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: access_timestamp: Sat Jul 10 09:39:05 2021 2021-07-10T09:44:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: modify_timestamp: Sat Jul 10 09:39:05 2021 2021-07-10T09:44:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: protected: False 2021-07-10T09:44:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring state: enabled 2021-07-10T09:44:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring mode: snapshot 2021-07-10T09:44:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring global id: f78b6b5a-849d-4b37-b86e-6e3d164b4c00 2021-07-10T09:44:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: mirroring primary: false 2021-07-10T09:44:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ compare_image_snaps mirror image_50 snap 2021-07-10T09:44:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:44:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap_name=snap 2021-07-10T09:44:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local ret=0 2021-07-10T09:44:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local rmt_export=/tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.export 2021-07-10T09:44:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local loc_export=/tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.export 2021-07-10T09:44:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.export 2021-07-10T09:44:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.export 2021-07-10T09:44:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.export 2021-07-10T09:44:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-07-10T09:44:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cmp /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.export 2021-07-10T09:44:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rm -f /tmp/tmp.VSH8FeaLBn/cluster2-mirror-image_50.export /tmp/tmp.VSH8FeaLBn/cluster1-mirror-image_50.export 2021-07-10T09:44:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ testlog 'TEST: delete many images' 2021-07-10T09:44:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee -a /tmp/tmp.VSH8FeaLBn/rbd-mirror.test.log 2021-07-10T09:44:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ date '+%F %T' 2021-07-10T09:44:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo 2021-07-10 09:44:56 TEST: delete many images 2021-07-10T09:44:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10 09:44:56 TEST: delete many images 2021-07-10T09:44:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 50 2021-07-10T09:44:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_1 2021-07-10T09:44:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2021-07-10T09:44:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:44:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:44:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2021-07-10T09:44:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:44:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_1 2021-07-10T09:44:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:44:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:44:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:44:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_1 2021-07-10T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2021-07-10T09:44:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2021-07-10T09:44:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:44:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:44:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:44:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_2 2021-07-10T09:44:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2021-07-10T09:44:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:44:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:44:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2021-07-10T09:44:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:44:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_2 2021-07-10T09:44:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:44:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:44:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:44:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_2 2021-07-10T09:44:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:44:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:44:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:44:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2021-07-10T09:44:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2021-07-10T09:45:00.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.873+0000 7f3185883700 -1 reset not still connected to 0x55912b5c1790 2021-07-10T09:45:00.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.873+0000 7f3185883700 -1 reset not still connected to 0x5591322000d0 2021-07-10T09:45:00.876 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x559134117040 2021-07-10T09:45:00.877 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x55913578cc30 2021-07-10T09:45:00.877 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x55913578dd40 2021-07-10T09:45:00.877 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x55913578de10 2021-07-10T09:45:00.877 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x55913d2b2d00 2021-07-10T09:45:00.878 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x559141bd9ad0 2021-07-10T09:45:00.878 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x559142f54000 2021-07-10T09:45:00.878 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:45:00.874+0000 7f3185883700 -1 reset not still connected to 0x559145f1cea0 2021-07-10T09:45:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_3 2021-07-10T09:45:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2021-07-10T09:45:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:45:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2021-07-10T09:45:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_3 2021-07-10T09:45:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:45:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_3 2021-07-10T09:45:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:45:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2021-07-10T09:45:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2021-07-10T09:45:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_4 2021-07-10T09:45:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2021-07-10T09:45:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:45:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2021-07-10T09:45:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_4 2021-07-10T09:45:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:45:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_4 2021-07-10T09:45:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:45:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2021-07-10T09:45:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2021-07-10T09:45:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_5 2021-07-10T09:45:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2021-07-10T09:45:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:45:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2021-07-10T09:45:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_5 2021-07-10T09:45:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:45:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_5 2021-07-10T09:45:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:45:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2021-07-10T09:45:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2021-07-10T09:45:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_6 2021-07-10T09:45:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2021-07-10T09:45:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:45:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2021-07-10T09:45:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_6 2021-07-10T09:45:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:45:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_6 2021-07-10T09:45:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:45:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2021-07-10T09:45:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2021-07-10T09:45:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_7 2021-07-10T09:45:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2021-07-10T09:45:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:45:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2021-07-10T09:45:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_7 2021-07-10T09:45:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:45:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_7 2021-07-10T09:45:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:45:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2021-07-10T09:45:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2021-07-10T09:45:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_8 2021-07-10T09:45:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2021-07-10T09:45:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:45:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2021-07-10T09:45:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_8 2021-07-10T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:45:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_8 2021-07-10T09:45:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:45:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2021-07-10T09:45:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2021-07-10T09:45:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_9 2021-07-10T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2021-07-10T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:45:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2021-07-10T09:45:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_9 2021-07-10T09:45:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:45:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_9 2021-07-10T09:45:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:45:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2021-07-10T09:45:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2021-07-10T09:45:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_10 2021-07-10T09:45:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2021-07-10T09:45:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:45:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2021-07-10T09:45:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_10 2021-07-10T09:45:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:45:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_10 2021-07-10T09:45:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:45:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2021-07-10T09:45:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2021-07-10T09:45:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_11 2021-07-10T09:45:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2021-07-10T09:45:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:45:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2021-07-10T09:45:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_11 2021-07-10T09:45:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:45:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_11 2021-07-10T09:45:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:45:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2021-07-10T09:45:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2021-07-10T09:45:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_12 2021-07-10T09:45:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2021-07-10T09:45:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:45:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2021-07-10T09:45:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_12 2021-07-10T09:45:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:45:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_12 2021-07-10T09:45:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:45:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2021-07-10T09:45:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2021-07-10T09:45:19.948 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:45:19.946+0000 7f69f674e700 -1 reset not still connected to 0x563079820270 2021-07-10T09:45:19.948 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:45:19.946+0000 7f69f674e700 -1 reset not still connected to 0x5630cbbc4750 2021-07-10T09:45:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_13 2021-07-10T09:45:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2021-07-10T09:45:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:45:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2021-07-10T09:45:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_13 2021-07-10T09:45:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:45:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_13 2021-07-10T09:45:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:45:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2021-07-10T09:45:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2021-07-10T09:45:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_14 2021-07-10T09:45:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2021-07-10T09:45:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2021-07-10T09:45:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_14 2021-07-10T09:45:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:45:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_14 2021-07-10T09:45:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:45:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2021-07-10T09:45:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2021-07-10T09:45:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_15 2021-07-10T09:45:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2021-07-10T09:45:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:45:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2021-07-10T09:45:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_15 2021-07-10T09:45:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:45:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_15 2021-07-10T09:45:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:45:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2021-07-10T09:45:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2021-07-10T09:45:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_16 2021-07-10T09:45:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2021-07-10T09:45:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:45:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2021-07-10T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_16 2021-07-10T09:45:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:45:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_16 2021-07-10T09:45:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:45:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2021-07-10T09:45:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2021-07-10T09:45:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_17 2021-07-10T09:45:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2021-07-10T09:45:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:45:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2021-07-10T09:45:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_17 2021-07-10T09:45:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:45:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_17 2021-07-10T09:45:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:45:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2021-07-10T09:45:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2021-07-10T09:45:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_18 2021-07-10T09:45:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2021-07-10T09:45:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:45:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2021-07-10T09:45:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_18 2021-07-10T09:45:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:45:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_18 2021-07-10T09:45:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:45:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2021-07-10T09:45:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2021-07-10T09:45:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_19 2021-07-10T09:45:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2021-07-10T09:45:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:45:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2021-07-10T09:45:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_19 2021-07-10T09:45:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:45:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_19 2021-07-10T09:45:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:45:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2021-07-10T09:45:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2021-07-10T09:45:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_20 2021-07-10T09:45:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2021-07-10T09:45:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:45:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2021-07-10T09:45:35.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7455785478587501) > weight (0.25) 2021-07-10T09:45:35.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6061668204576851) > weight (0.25) 2021-07-10T09:45:35.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.928094708981267) > weight (0.25) 2021-07-10T09:45:35.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8143944826639705) > weight (0.25) 2021-07-10T09:45:35.724 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2021-07-10T09:45:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_20 2021-07-10T09:45:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:45:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_20 2021-07-10T09:45:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:45:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2021-07-10T09:45:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2021-07-10T09:45:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_21 2021-07-10T09:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2021-07-10T09:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:45:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2021-07-10T09:45:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_21 2021-07-10T09:45:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:45:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_21 2021-07-10T09:45:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:45:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2021-07-10T09:45:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2021-07-10T09:45:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_22 2021-07-10T09:45:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2021-07-10T09:45:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:45:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2021-07-10T09:45:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_22 2021-07-10T09:45:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:45:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_22 2021-07-10T09:45:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:45:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2021-07-10T09:45:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2021-07-10T09:45:40.732 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:45:40.729+0000 7f69f674e700 -1 reset not still connected to 0x5630789b8820 2021-07-10T09:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_23 2021-07-10T09:45:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2021-07-10T09:45:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:45:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2021-07-10T09:45:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_23 2021-07-10T09:45:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:45:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_23 2021-07-10T09:45:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:45:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2021-07-10T09:45:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2021-07-10T09:45:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_24 2021-07-10T09:45:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2021-07-10T09:45:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:45:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2021-07-10T09:45:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_24 2021-07-10T09:45:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:45:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_24 2021-07-10T09:45:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:45:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2021-07-10T09:45:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2021-07-10T09:45:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_25 2021-07-10T09:45:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2021-07-10T09:45:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:45:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2021-07-10T09:45:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_25 2021-07-10T09:45:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:45:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_25 2021-07-10T09:45:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:45:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2021-07-10T09:45:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2021-07-10T09:45:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_26 2021-07-10T09:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2021-07-10T09:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:45:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2021-07-10T09:45:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_26 2021-07-10T09:45:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:45:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_26 2021-07-10T09:45:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:45:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2021-07-10T09:45:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2021-07-10T09:45:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_27 2021-07-10T09:45:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2021-07-10T09:45:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:45:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2021-07-10T09:45:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_27 2021-07-10T09:45:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:45:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_27 2021-07-10T09:45:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:45:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2021-07-10T09:45:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2021-07-10T09:45:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_28 2021-07-10T09:45:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2021-07-10T09:45:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:45:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2021-07-10T09:45:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_28 2021-07-10T09:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_28 2021-07-10T09:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2021-07-10T09:45:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2021-07-10T09:45:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_29 2021-07-10T09:45:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2021-07-10T09:45:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:45:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2021-07-10T09:45:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_29 2021-07-10T09:45:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:45:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_29 2021-07-10T09:45:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:45:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2021-07-10T09:45:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2021-07-10T09:45:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_30 2021-07-10T09:45:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2021-07-10T09:45:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:45:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2021-07-10T09:45:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_30 2021-07-10T09:45:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:45:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_30 2021-07-10T09:45:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:45:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2021-07-10T09:45:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2021-07-10T09:45:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_31 2021-07-10T09:45:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2021-07-10T09:45:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:45:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2021-07-10T09:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_31 2021-07-10T09:45:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:45:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_31 2021-07-10T09:45:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:45:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2021-07-10T09:45:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2021-07-10T09:45:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:45:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:45:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:45:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_32 2021-07-10T09:45:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2021-07-10T09:45:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:45:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:45:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2021-07-10T09:45:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:45:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_32 2021-07-10T09:45:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:45:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:45:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:45:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_32 2021-07-10T09:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2021-07-10T09:46:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2021-07-10T09:46:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_33 2021-07-10T09:46:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2021-07-10T09:46:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:46:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2021-07-10T09:46:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_33 2021-07-10T09:46:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:46:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_33 2021-07-10T09:46:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:46:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2021-07-10T09:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2021-07-10T09:46:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_34 2021-07-10T09:46:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2021-07-10T09:46:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:46:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2021-07-10T09:46:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_34 2021-07-10T09:46:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:46:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_34 2021-07-10T09:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:46:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2021-07-10T09:46:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2021-07-10T09:46:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_35 2021-07-10T09:46:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2021-07-10T09:46:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:46:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2021-07-10T09:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_35 2021-07-10T09:46:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:46:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_35 2021-07-10T09:46:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:46:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2021-07-10T09:46:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2021-07-10T09:46:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_36 2021-07-10T09:46:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2021-07-10T09:46:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:46:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2021-07-10T09:46:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_36 2021-07-10T09:46:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:46:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_36 2021-07-10T09:46:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:46:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2021-07-10T09:46:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2021-07-10T09:46:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_37 2021-07-10T09:46:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2021-07-10T09:46:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:46:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2021-07-10T09:46:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_37 2021-07-10T09:46:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:46:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_37 2021-07-10T09:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:46:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2021-07-10T09:46:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2021-07-10T09:46:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_38 2021-07-10T09:46:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2021-07-10T09:46:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:46:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2021-07-10T09:46:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_38 2021-07-10T09:46:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:46:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_38 2021-07-10T09:46:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:46:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2021-07-10T09:46:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2021-07-10T09:46:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_39 2021-07-10T09:46:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2021-07-10T09:46:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:46:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2021-07-10T09:46:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_39 2021-07-10T09:46:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:46:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_39 2021-07-10T09:46:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:46:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2021-07-10T09:46:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2021-07-10T09:46:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_40 2021-07-10T09:46:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2021-07-10T09:46:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:46:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2021-07-10T09:46:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_40 2021-07-10T09:46:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:46:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_40 2021-07-10T09:46:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:46:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2021-07-10T09:46:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2021-07-10T09:46:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_41 2021-07-10T09:46:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2021-07-10T09:46:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:46:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2021-07-10T09:46:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_41 2021-07-10T09:46:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:46:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_41 2021-07-10T09:46:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:46:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2021-07-10T09:46:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2021-07-10T09:46:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_42 2021-07-10T09:46:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2021-07-10T09:46:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:46:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2021-07-10T09:46:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_42 2021-07-10T09:46:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:46:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_42 2021-07-10T09:46:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:46:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2021-07-10T09:46:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2021-07-10T09:46:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_43 2021-07-10T09:46:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2021-07-10T09:46:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:46:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2021-07-10T09:46:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_43 2021-07-10T09:46:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:46:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_43 2021-07-10T09:46:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:46:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2021-07-10T09:46:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2021-07-10T09:46:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_44 2021-07-10T09:46:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2021-07-10T09:46:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:46:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2021-07-10T09:46:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_44 2021-07-10T09:46:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:46:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_44 2021-07-10T09:46:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:46:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2021-07-10T09:46:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2021-07-10T09:46:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_45 2021-07-10T09:46:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2021-07-10T09:46:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:46:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2021-07-10T09:46:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_45 2021-07-10T09:46:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:46:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_45 2021-07-10T09:46:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:46:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2021-07-10T09:46:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2021-07-10T09:46:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_46 2021-07-10T09:46:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2021-07-10T09:46:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:46:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2021-07-10T09:46:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_46 2021-07-10T09:46:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:46:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_46 2021-07-10T09:46:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:46:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2021-07-10T09:46:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2021-07-10T09:46:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_47 2021-07-10T09:46:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2021-07-10T09:46:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:46:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2021-07-10T09:46:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_47 2021-07-10T09:46:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:46:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_47 2021-07-10T09:46:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:46:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2021-07-10T09:46:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2021-07-10T09:46:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_48 2021-07-10T09:46:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2021-07-10T09:46:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:46:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2021-07-10T09:46:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_48 2021-07-10T09:46:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:46:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_48 2021-07-10T09:46:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2021-07-10T09:46:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2021-07-10T09:46:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_49 2021-07-10T09:46:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2021-07-10T09:46:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:46:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2021-07-10T09:46:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_49 2021-07-10T09:46:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:46:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_49 2021-07-10T09:46:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:46:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2021-07-10T09:46:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2021-07-10T09:46:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:46:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_50 2021-07-10T09:46:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2021-07-10T09:46:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:46:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local snap=snap 2021-07-10T09:46:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2021-07-10T09:46:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing snap: 100% complete...done. 2021-07-10T09:46:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_50 2021-07-10T09:46:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:46:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:46:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:46:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_50 2021-07-10T09:46:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:46:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:46:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2021-07-10T09:46:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2021-07-10T09:46:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-07-10T09:46:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:46:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ testlog 'TEST: image deletions should propagate' 2021-07-10T09:46:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee -a /tmp/tmp.VSH8FeaLBn/rbd-mirror.test.log 2021-07-10T09:46:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ date '+%F %T' 2021-07-10T09:46:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo 2021-07-10 09:46:37 TEST: image deletions should propagate 2021-07-10T09:46:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10 09:46:37 TEST: image deletions should propagate 2021-07-10T09:46:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_pool_images cluster1 mirror 0 2021-07-10T09:46:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:46:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:46:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_count=0 2021-07-10T09:46:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:46:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local count 2021-07-10T09:46:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local last_count=0 2021-07-10T09:46:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:46:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 40 2021-07-10T09:46:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:46:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 1 -ne 1 2021-07-10T09:46:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2021-07-10T09:46:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'images: ' 2021-07-10T09:46:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 2 2021-07-10T09:46:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:46:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ count=12 2021-07-10T09:46:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 12 = 0 2021-07-10T09:46:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 12 -ne 0 2021-07-10T09:46:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ break 2021-07-10T09:46:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 12 -eq 0 2021-07-10T09:46:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ last_count=12 2021-07-10T09:46:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ true 2021-07-10T09:46:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 40 2021-07-10T09:46:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:46:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 1 -ne 1 2021-07-10T09:46:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2021-07-10T09:46:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'images: ' 2021-07-10T09:46:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 2 2021-07-10T09:46:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:46:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ count=12 2021-07-10T09:46:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 12 = 0 2021-07-10T09:46:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 12 -ne 12 2021-07-10T09:46:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:46:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 2 -ne 1 2021-07-10T09:46:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 30 2021-07-10T09:46:48.052 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:46:48.050+0000 7f72d78b2700 -1 reset not still connected to 0x55dd22b476c0 2021-07-10T09:47:00.015 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:47:00.012+0000 7f4236192700 -1 reset not still connected to 0x55a45b815790 2021-07-10T09:47:00.017 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:47:00.015+0000 7f4236192700 -1 reset not still connected to 0x55a46f37e340 2021-07-10T09:47:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2021-07-10T09:47:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'images: ' 2021-07-10T09:47:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 2 2021-07-10T09:47:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:47:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ count=0 2021-07-10T09:47:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 0 = 0 2021-07-10T09:47:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2021-07-10T09:47:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state 2021-07-10T09:47:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 40 2021-07-10T09:47:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in `seq 1 40` 2021-07-10T09:47:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test 1 -ne 1 2021-07-10T09:47:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2021-07-10T09:47:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ grep 'image health:' 2021-07-10T09:47:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cut '-d ' -f 3 2021-07-10T09:47:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: failed to query services: (13) Permission denied 2021-07-10T09:47:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ state=OK 2021-07-10T09:47:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test OK = ERROR 2021-07-10T09:47:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test OK = OK 2021-07-10T09:47:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 50 2021-07-10T09:47:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_1 2021-07-10T09:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2021-07-10T09:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_1 2021-07-10T09:47:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_1 2021-07-10T09:47:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2021-07-10T09:47:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_1: (2) No such file or directory 2021-07-10T09:47:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2021-07-10T09:47:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_1 2021-07-10T09:47:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_1 2021-07-10T09:47:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2021-07-10T09:47:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_1: (2) No such file or directory 2021-07-10T09:47:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_2 2021-07-10T09:47:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2021-07-10T09:47:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:47:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_2 2021-07-10T09:47:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_2 2021-07-10T09:47:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2021-07-10T09:47:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_2: (2) No such file or directory 2021-07-10T09:47:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2021-07-10T09:47:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:47:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_2 2021-07-10T09:47:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_2 2021-07-10T09:47:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2021-07-10T09:47:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_2: (2) No such file or directory 2021-07-10T09:47:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_3 2021-07-10T09:47:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2021-07-10T09:47:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:47:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_3 2021-07-10T09:47:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_3 2021-07-10T09:47:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2021-07-10T09:47:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_3: (2) No such file or directory 2021-07-10T09:47:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2021-07-10T09:47:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:47:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_3 2021-07-10T09:47:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_3 2021-07-10T09:47:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2021-07-10T09:47:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_3: (2) No such file or directory 2021-07-10T09:47:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_4 2021-07-10T09:47:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2021-07-10T09:47:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:47:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_4 2021-07-10T09:47:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_4 2021-07-10T09:47:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2021-07-10T09:47:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_4: (2) No such file or directory 2021-07-10T09:47:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2021-07-10T09:47:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:47:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_4 2021-07-10T09:47:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_4 2021-07-10T09:47:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2021-07-10T09:47:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_4: (2) No such file or directory 2021-07-10T09:47:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_5 2021-07-10T09:47:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2021-07-10T09:47:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:47:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_5 2021-07-10T09:47:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_5 2021-07-10T09:47:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2021-07-10T09:47:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_5: (2) No such file or directory 2021-07-10T09:47:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2021-07-10T09:47:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:47:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_5 2021-07-10T09:47:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_5 2021-07-10T09:47:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2021-07-10T09:47:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_5: (2) No such file or directory 2021-07-10T09:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_6 2021-07-10T09:47:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2021-07-10T09:47:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:47:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_6 2021-07-10T09:47:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_6 2021-07-10T09:47:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2021-07-10T09:47:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_6: (2) No such file or directory 2021-07-10T09:47:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2021-07-10T09:47:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:47:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_6 2021-07-10T09:47:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_6 2021-07-10T09:47:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2021-07-10T09:47:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_6: (2) No such file or directory 2021-07-10T09:47:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_7 2021-07-10T09:47:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2021-07-10T09:47:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:47:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_7 2021-07-10T09:47:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_7 2021-07-10T09:47:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2021-07-10T09:47:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_7: (2) No such file or directory 2021-07-10T09:47:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2021-07-10T09:47:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:47:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_7 2021-07-10T09:47:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_7 2021-07-10T09:47:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2021-07-10T09:47:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_7: (2) No such file or directory 2021-07-10T09:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_8 2021-07-10T09:47:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2021-07-10T09:47:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:47:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_8 2021-07-10T09:47:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_8 2021-07-10T09:47:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2021-07-10T09:47:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_8: (2) No such file or directory 2021-07-10T09:47:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2021-07-10T09:47:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:47:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_8 2021-07-10T09:47:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_8 2021-07-10T09:47:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2021-07-10T09:47:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_8: (2) No such file or directory 2021-07-10T09:47:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_9 2021-07-10T09:47:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2021-07-10T09:47:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:47:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T09:47:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T09:47:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T09:47:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_9: (2) No such file or directory 2021-07-10T09:47:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2021-07-10T09:47:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:47:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T09:47:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T09:47:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T09:47:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_9: (2) No such file or directory 2021-07-10T09:47:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_10 2021-07-10T09:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2021-07-10T09:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_10 2021-07-10T09:47:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_10 2021-07-10T09:47:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2021-07-10T09:47:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_10: (2) No such file or directory 2021-07-10T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2021-07-10T09:47:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_10 2021-07-10T09:47:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_10 2021-07-10T09:47:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_10 2021-07-10T09:47:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2021-07-10T09:47:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_10: (2) No such file or directory 2021-07-10T09:47:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_11 2021-07-10T09:47:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2021-07-10T09:47:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:47:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_11 2021-07-10T09:47:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_11 2021-07-10T09:47:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2021-07-10T09:47:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_11: (2) No such file or directory 2021-07-10T09:47:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2021-07-10T09:47:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_11 2021-07-10T09:47:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_11 2021-07-10T09:47:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_11 2021-07-10T09:47:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2021-07-10T09:47:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_11: (2) No such file or directory 2021-07-10T09:47:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_12 2021-07-10T09:47:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2021-07-10T09:47:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:47:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_12 2021-07-10T09:47:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_12 2021-07-10T09:47:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2021-07-10T09:47:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_12: (2) No such file or directory 2021-07-10T09:47:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2021-07-10T09:47:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_12 2021-07-10T09:47:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_12 2021-07-10T09:47:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_12 2021-07-10T09:47:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2021-07-10T09:47:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_12: (2) No such file or directory 2021-07-10T09:47:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_13 2021-07-10T09:47:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2021-07-10T09:47:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:47:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_13 2021-07-10T09:47:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_13 2021-07-10T09:47:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2021-07-10T09:47:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_13: (2) No such file or directory 2021-07-10T09:47:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2021-07-10T09:47:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_13 2021-07-10T09:47:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_13 2021-07-10T09:47:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_13 2021-07-10T09:47:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2021-07-10T09:47:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_13: (2) No such file or directory 2021-07-10T09:47:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_14 2021-07-10T09:47:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2021-07-10T09:47:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:47:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_14 2021-07-10T09:47:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_14 2021-07-10T09:47:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2021-07-10T09:47:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_14: (2) No such file or directory 2021-07-10T09:47:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2021-07-10T09:47:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_14 2021-07-10T09:47:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_14 2021-07-10T09:47:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_14 2021-07-10T09:47:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2021-07-10T09:47:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_14: (2) No such file or directory 2021-07-10T09:47:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_15 2021-07-10T09:47:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2021-07-10T09:47:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:47:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_15 2021-07-10T09:47:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_15 2021-07-10T09:47:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2021-07-10T09:47:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_15: (2) No such file or directory 2021-07-10T09:47:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2021-07-10T09:47:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_15 2021-07-10T09:47:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_15 2021-07-10T09:47:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_15 2021-07-10T09:47:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2021-07-10T09:47:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_15: (2) No such file or directory 2021-07-10T09:47:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_16 2021-07-10T09:47:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2021-07-10T09:47:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:47:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_16 2021-07-10T09:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_16 2021-07-10T09:47:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2021-07-10T09:47:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_16: (2) No such file or directory 2021-07-10T09:47:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2021-07-10T09:47:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_16 2021-07-10T09:47:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_16 2021-07-10T09:47:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_16 2021-07-10T09:47:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2021-07-10T09:47:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_16: (2) No such file or directory 2021-07-10T09:47:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_17 2021-07-10T09:47:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2021-07-10T09:47:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:47:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_17 2021-07-10T09:47:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_17 2021-07-10T09:47:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2021-07-10T09:47:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_17: (2) No such file or directory 2021-07-10T09:47:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2021-07-10T09:47:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_17 2021-07-10T09:47:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_17 2021-07-10T09:47:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_17 2021-07-10T09:47:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2021-07-10T09:47:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_17: (2) No such file or directory 2021-07-10T09:47:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_18 2021-07-10T09:47:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2021-07-10T09:47:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:47:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_18 2021-07-10T09:47:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_18 2021-07-10T09:47:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2021-07-10T09:47:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_18: (2) No such file or directory 2021-07-10T09:47:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2021-07-10T09:47:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_18 2021-07-10T09:47:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_18 2021-07-10T09:47:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_18 2021-07-10T09:47:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2021-07-10T09:47:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_18: (2) No such file or directory 2021-07-10T09:47:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_19 2021-07-10T09:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2021-07-10T09:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_19 2021-07-10T09:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_19 2021-07-10T09:47:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2021-07-10T09:47:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_19: (2) No such file or directory 2021-07-10T09:47:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2021-07-10T09:47:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_19 2021-07-10T09:47:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_19 2021-07-10T09:47:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_19 2021-07-10T09:47:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2021-07-10T09:47:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_19: (2) No such file or directory 2021-07-10T09:47:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_20 2021-07-10T09:47:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2021-07-10T09:47:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:47:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_20 2021-07-10T09:47:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_20 2021-07-10T09:47:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2021-07-10T09:47:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_20: (2) No such file or directory 2021-07-10T09:47:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2021-07-10T09:47:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_20 2021-07-10T09:47:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_20 2021-07-10T09:47:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_20 2021-07-10T09:47:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2021-07-10T09:47:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_20: (2) No such file or directory 2021-07-10T09:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_21 2021-07-10T09:47:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2021-07-10T09:47:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:47:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_21 2021-07-10T09:47:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_21 2021-07-10T09:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2021-07-10T09:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_21: (2) No such file or directory 2021-07-10T09:47:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2021-07-10T09:47:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_21 2021-07-10T09:47:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_21 2021-07-10T09:47:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_21 2021-07-10T09:47:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2021-07-10T09:47:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_21: (2) No such file or directory 2021-07-10T09:47:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_22 2021-07-10T09:47:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2021-07-10T09:47:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:47:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_22 2021-07-10T09:47:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_22 2021-07-10T09:47:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2021-07-10T09:47:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_22: (2) No such file or directory 2021-07-10T09:47:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2021-07-10T09:47:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_22 2021-07-10T09:47:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_22 2021-07-10T09:47:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_22 2021-07-10T09:47:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2021-07-10T09:47:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_22: (2) No such file or directory 2021-07-10T09:47:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_23 2021-07-10T09:47:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2021-07-10T09:47:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:47:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_23 2021-07-10T09:47:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_23 2021-07-10T09:47:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2021-07-10T09:47:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_23: (2) No such file or directory 2021-07-10T09:47:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2021-07-10T09:47:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_23 2021-07-10T09:47:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_23 2021-07-10T09:47:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_23 2021-07-10T09:47:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2021-07-10T09:47:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_23: (2) No such file or directory 2021-07-10T09:47:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_24 2021-07-10T09:47:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2021-07-10T09:47:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:47:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_24 2021-07-10T09:47:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_24 2021-07-10T09:47:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2021-07-10T09:47:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_24: (2) No such file or directory 2021-07-10T09:47:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2021-07-10T09:47:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_24 2021-07-10T09:47:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_24 2021-07-10T09:47:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_24 2021-07-10T09:47:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2021-07-10T09:47:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_24: (2) No such file or directory 2021-07-10T09:47:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_25 2021-07-10T09:47:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2021-07-10T09:47:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:47:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_25 2021-07-10T09:47:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_25 2021-07-10T09:47:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2021-07-10T09:47:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_25: (2) No such file or directory 2021-07-10T09:47:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2021-07-10T09:47:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_25 2021-07-10T09:47:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_25 2021-07-10T09:47:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_25 2021-07-10T09:47:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2021-07-10T09:47:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_25: (2) No such file or directory 2021-07-10T09:47:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_26 2021-07-10T09:47:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2021-07-10T09:47:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:47:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_26 2021-07-10T09:47:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_26 2021-07-10T09:47:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2021-07-10T09:47:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_26: (2) No such file or directory 2021-07-10T09:47:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2021-07-10T09:47:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_26 2021-07-10T09:47:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_26 2021-07-10T09:47:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_26 2021-07-10T09:47:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2021-07-10T09:47:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_26: (2) No such file or directory 2021-07-10T09:47:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_27 2021-07-10T09:47:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2021-07-10T09:47:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:47:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_27 2021-07-10T09:47:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_27 2021-07-10T09:47:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2021-07-10T09:47:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_27: (2) No such file or directory 2021-07-10T09:47:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2021-07-10T09:47:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_27 2021-07-10T09:47:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_27 2021-07-10T09:47:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_27 2021-07-10T09:47:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2021-07-10T09:47:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_27: (2) No such file or directory 2021-07-10T09:47:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_28 2021-07-10T09:47:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2021-07-10T09:47:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:47:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_28 2021-07-10T09:47:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_28 2021-07-10T09:47:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2021-07-10T09:47:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_28: (2) No such file or directory 2021-07-10T09:47:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2021-07-10T09:47:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_28 2021-07-10T09:47:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_28 2021-07-10T09:47:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_28 2021-07-10T09:47:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2021-07-10T09:47:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_28: (2) No such file or directory 2021-07-10T09:47:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_29 2021-07-10T09:47:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2021-07-10T09:47:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:47:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_29 2021-07-10T09:47:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_29 2021-07-10T09:47:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2021-07-10T09:47:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_29: (2) No such file or directory 2021-07-10T09:47:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2021-07-10T09:47:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_29 2021-07-10T09:47:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_29 2021-07-10T09:47:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_29 2021-07-10T09:47:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2021-07-10T09:47:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_29: (2) No such file or directory 2021-07-10T09:47:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_30 2021-07-10T09:47:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2021-07-10T09:47:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:47:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_30 2021-07-10T09:47:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_30 2021-07-10T09:47:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2021-07-10T09:47:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_30: (2) No such file or directory 2021-07-10T09:47:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2021-07-10T09:47:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_30 2021-07-10T09:47:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_30 2021-07-10T09:47:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_30 2021-07-10T09:47:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2021-07-10T09:47:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_30: (2) No such file or directory 2021-07-10T09:47:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_31 2021-07-10T09:47:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2021-07-10T09:47:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:47:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_31 2021-07-10T09:47:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_31 2021-07-10T09:47:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2021-07-10T09:47:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_31: (2) No such file or directory 2021-07-10T09:47:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2021-07-10T09:47:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_31 2021-07-10T09:47:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_31 2021-07-10T09:47:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_31 2021-07-10T09:47:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2021-07-10T09:47:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_31: (2) No such file or directory 2021-07-10T09:47:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_32 2021-07-10T09:47:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2021-07-10T09:47:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:47:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_32 2021-07-10T09:47:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_32 2021-07-10T09:47:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2021-07-10T09:47:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_32: (2) No such file or directory 2021-07-10T09:47:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2021-07-10T09:47:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_32 2021-07-10T09:47:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_32 2021-07-10T09:47:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_32 2021-07-10T09:47:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2021-07-10T09:47:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_32: (2) No such file or directory 2021-07-10T09:47:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_33 2021-07-10T09:47:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2021-07-10T09:47:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:47:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_33 2021-07-10T09:47:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_33 2021-07-10T09:47:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2021-07-10T09:47:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_33: (2) No such file or directory 2021-07-10T09:47:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2021-07-10T09:47:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_33 2021-07-10T09:47:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_33 2021-07-10T09:47:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_33 2021-07-10T09:47:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2021-07-10T09:47:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_33: (2) No such file or directory 2021-07-10T09:47:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_34 2021-07-10T09:47:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2021-07-10T09:47:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:47:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_34 2021-07-10T09:47:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_34 2021-07-10T09:47:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2021-07-10T09:47:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_34: (2) No such file or directory 2021-07-10T09:47:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2021-07-10T09:47:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_34 2021-07-10T09:47:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_34 2021-07-10T09:47:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_34 2021-07-10T09:47:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2021-07-10T09:47:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_34: (2) No such file or directory 2021-07-10T09:47:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_35 2021-07-10T09:47:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2021-07-10T09:47:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:47:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_35 2021-07-10T09:47:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_35 2021-07-10T09:47:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2021-07-10T09:47:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_35: (2) No such file or directory 2021-07-10T09:47:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2021-07-10T09:47:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_35 2021-07-10T09:47:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_35 2021-07-10T09:47:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_35 2021-07-10T09:47:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2021-07-10T09:47:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_35: (2) No such file or directory 2021-07-10T09:47:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_36 2021-07-10T09:47:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2021-07-10T09:47:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:47:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_36 2021-07-10T09:47:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_36 2021-07-10T09:47:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2021-07-10T09:47:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_36: (2) No such file or directory 2021-07-10T09:47:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2021-07-10T09:47:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_36 2021-07-10T09:47:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_36 2021-07-10T09:47:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_36 2021-07-10T09:47:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2021-07-10T09:47:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_36: (2) No such file or directory 2021-07-10T09:47:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_37 2021-07-10T09:47:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2021-07-10T09:47:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:47:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_37 2021-07-10T09:47:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_37 2021-07-10T09:47:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2021-07-10T09:47:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_37: (2) No such file or directory 2021-07-10T09:47:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2021-07-10T09:47:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_37 2021-07-10T09:47:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_37 2021-07-10T09:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_37 2021-07-10T09:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2021-07-10T09:47:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_37: (2) No such file or directory 2021-07-10T09:47:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_38 2021-07-10T09:47:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2021-07-10T09:47:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:47:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_38 2021-07-10T09:47:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_38 2021-07-10T09:47:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2021-07-10T09:47:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_38: (2) No such file or directory 2021-07-10T09:47:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2021-07-10T09:47:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_38 2021-07-10T09:47:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_38 2021-07-10T09:47:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_38 2021-07-10T09:47:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2021-07-10T09:47:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_38: (2) No such file or directory 2021-07-10T09:47:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_39 2021-07-10T09:47:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2021-07-10T09:47:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:47:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_39 2021-07-10T09:47:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_39 2021-07-10T09:47:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2021-07-10T09:47:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_39: (2) No such file or directory 2021-07-10T09:47:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2021-07-10T09:47:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_39 2021-07-10T09:47:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_39 2021-07-10T09:47:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_39 2021-07-10T09:47:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2021-07-10T09:47:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_39: (2) No such file or directory 2021-07-10T09:47:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_40 2021-07-10T09:47:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2021-07-10T09:47:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:47:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_40 2021-07-10T09:47:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_40 2021-07-10T09:47:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2021-07-10T09:47:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_40: (2) No such file or directory 2021-07-10T09:47:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2021-07-10T09:47:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_40 2021-07-10T09:47:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_40 2021-07-10T09:47:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_40 2021-07-10T09:47:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2021-07-10T09:47:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_40: (2) No such file or directory 2021-07-10T09:47:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_41 2021-07-10T09:47:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2021-07-10T09:47:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:47:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_41 2021-07-10T09:47:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_41 2021-07-10T09:47:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2021-07-10T09:47:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_41: (2) No such file or directory 2021-07-10T09:47:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2021-07-10T09:47:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_41 2021-07-10T09:47:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_41 2021-07-10T09:47:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_41 2021-07-10T09:47:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2021-07-10T09:47:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_41: (2) No such file or directory 2021-07-10T09:47:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_42 2021-07-10T09:47:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2021-07-10T09:47:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:47:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_42 2021-07-10T09:47:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_42 2021-07-10T09:47:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2021-07-10T09:47:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_42: (2) No such file or directory 2021-07-10T09:47:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2021-07-10T09:47:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_42 2021-07-10T09:47:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_42 2021-07-10T09:47:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_42 2021-07-10T09:47:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2021-07-10T09:47:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_42: (2) No such file or directory 2021-07-10T09:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_43 2021-07-10T09:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2021-07-10T09:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:47:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_43 2021-07-10T09:47:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_43 2021-07-10T09:47:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2021-07-10T09:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_43: (2) No such file or directory 2021-07-10T09:47:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2021-07-10T09:47:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_43 2021-07-10T09:47:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_43 2021-07-10T09:47:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_43 2021-07-10T09:47:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2021-07-10T09:47:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_43: (2) No such file or directory 2021-07-10T09:47:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_44 2021-07-10T09:47:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2021-07-10T09:47:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:47:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_44 2021-07-10T09:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_44 2021-07-10T09:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2021-07-10T09:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_44: (2) No such file or directory 2021-07-10T09:47:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2021-07-10T09:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_44 2021-07-10T09:47:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_44 2021-07-10T09:47:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_44 2021-07-10T09:47:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2021-07-10T09:47:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_44: (2) No such file or directory 2021-07-10T09:47:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_45 2021-07-10T09:47:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2021-07-10T09:47:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:47:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_45 2021-07-10T09:47:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_45 2021-07-10T09:47:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2021-07-10T09:47:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_45: (2) No such file or directory 2021-07-10T09:47:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2021-07-10T09:47:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_45 2021-07-10T09:47:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_45 2021-07-10T09:47:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_45 2021-07-10T09:47:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2021-07-10T09:47:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_45: (2) No such file or directory 2021-07-10T09:47:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_46 2021-07-10T09:47:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2021-07-10T09:47:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_46 2021-07-10T09:47:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_46 2021-07-10T09:47:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2021-07-10T09:47:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_46: (2) No such file or directory 2021-07-10T09:47:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2021-07-10T09:47:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_46 2021-07-10T09:47:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_46 2021-07-10T09:47:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_46 2021-07-10T09:47:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2021-07-10T09:47:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_46: (2) No such file or directory 2021-07-10T09:47:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_47 2021-07-10T09:47:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2021-07-10T09:47:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:47:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_47 2021-07-10T09:47:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_47 2021-07-10T09:47:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2021-07-10T09:47:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_47: (2) No such file or directory 2021-07-10T09:47:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2021-07-10T09:47:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_47 2021-07-10T09:47:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_47 2021-07-10T09:47:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_47 2021-07-10T09:47:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2021-07-10T09:47:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_47: (2) No such file or directory 2021-07-10T09:47:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_48 2021-07-10T09:47:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2021-07-10T09:47:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:47:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_48 2021-07-10T09:47:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_48 2021-07-10T09:47:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2021-07-10T09:47:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_48: (2) No such file or directory 2021-07-10T09:47:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2021-07-10T09:47:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_48 2021-07-10T09:47:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_48 2021-07-10T09:47:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_48 2021-07-10T09:47:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2021-07-10T09:47:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_48: (2) No such file or directory 2021-07-10T09:47:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_49 2021-07-10T09:47:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2021-07-10T09:47:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:47:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_49 2021-07-10T09:47:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_49 2021-07-10T09:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2021-07-10T09:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_49: (2) No such file or directory 2021-07-10T09:47:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2021-07-10T09:47:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_49 2021-07-10T09:47:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_49 2021-07-10T09:47:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_49 2021-07-10T09:47:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2021-07-10T09:47:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_49: (2) No such file or directory 2021-07-10T09:47:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2021-07-10T09:47:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_50 2021-07-10T09:47:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2021-07-10T09:47:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:47:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_50 2021-07-10T09:47:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_50 2021-07-10T09:47:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2021-07-10T09:47:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_50: (2) No such file or directory 2021-07-10T09:47:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2021-07-10T09:47:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_50 2021-07-10T09:47:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_50 2021-07-10T09:47:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_50 2021-07-10T09:47:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2021-07-10T09:47:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_50: (2) No such file or directory 2021-07-10T09:47:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ testlog 'TEST: delete images during bootstrap' 2021-07-10T09:47:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ tee -a /tmp/tmp.VSH8FeaLBn/rbd-mirror.test.log 2021-07-10T09:47:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ date '+%F %T' 2021-07-10T09:47:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ echo 2021-07-10 09:47:15 TEST: delete images during bootstrap 2021-07-10T09:47:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10 09:47:15 TEST: delete images during bootstrap 2021-07-10T09:47:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_pool_mirror_mode cluster1 mirror image 2021-07-10T09:47:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=image 2021-07-10T09:47:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2021-07-10T09:47:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd: mirroring is already configured for image mode 2021-07-10T09:47:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_pool_mirror_mode cluster2 mirror image 2021-07-10T09:47:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=image 2021-07-10T09:47:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2021-07-10T09:47:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:rbd: mirroring is already configured for image mode 2021-07-10T09:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ start_mirror cluster1 2021-07-10T09:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance 2021-07-10T09:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set_cluster_instance cluster1 cluster instance 2021-07-10T09:47:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local val=cluster1 2021-07-10T09:47:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster_var_name=cluster 2021-07-10T09:47:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local instance_var_name=instance 2021-07-10T09:47:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ cluster=cluster1 2021-07-10T09:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=cluster1 2021-07-10T09:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' cluster1 = cluster1 ']' 2021-07-10T09:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ instance=0 2021-07-10T09:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval cluster=cluster1 2021-07-10T09:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cluster=cluster1 2021-07-10T09:47:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ eval instance=0 2021-07-10T09:47:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ instance=0 2021-07-10T09:47:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 1 2021-07-10T09:47:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 2021-07-10T09:47:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=test 2021-07-10T09:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ seq 1 10 2021-07-10T09:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_1 2021-07-10T09:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_1 512M 2021-07-10T09:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:47:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:47:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:47:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:47:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:47:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:47:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:47:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_1 2021-07-10T09:47:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_1 2021-07-10T09:47:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2021-07-10T09:47:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:47:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_1 2021-07-10T09:47:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:47:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:47:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:47:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:47:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 15724544 bytes 2021-07-10T09:47:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 44145664 bytes 2021-07-10T09:47:17.625 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:47:17.623+0000 7f2574cc7700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:47:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 133209088 bytes 2021-07-10T09:47:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 170510336 bytes 2021-07-10T09:47:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 207810560 bytes 2021-07-10T09:47:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 261469184 bytes 2021-07-10T09:47:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 283090944 bytes 2021-07-10T09:47:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 294089728 bytes 2021-07-10T09:47:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 342900736 bytes 2021-07-10T09:47:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 380687360 bytes 2021-07-10T09:47:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 390715392 bytes 2021-07-10T09:47:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 456396800 bytes 2021-07-10T09:47:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 498164736 bytes 2021-07-10T09:47:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 534575104 bytes 2021-07-10T09:47:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 582682624 bytes 2021-07-10T09:47:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 643480576 bytes 2021-07-10T09:47:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 674010112 bytes 2021-07-10T09:47:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 712311808 bytes 2021-07-10T09:47:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 714614784 bytes 2021-07-10T09:47:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 754121728 bytes 2021-07-10T09:47:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 784142336 bytes 2021-07-10T09:47:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 790751232 bytes 2021-07-10T09:47:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 804783104 bytes 2021-07-10T09:47:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 822256640 bytes 2021-07-10T09:47:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 840781824 bytes 2021-07-10T09:47:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 903623680 bytes 2021-07-10T09:47:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 980121600 bytes 2021-07-10T09:47:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1018631168 bytes 2021-07-10T09:47:24.726 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8593536439944435) > weight (0.25) 2021-07-10T09:47:24.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.30869183898665986) > weight (0.25) 2021-07-10T09:47:24.727 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2021-07-10T09:47:24.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2021-07-10T09:47:24.728 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2021-07-10T09:47:24.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi105.stderr:2021-07-10T09:47:24.729+0000 7f2bf2565700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42965) UID: 1000 2021-07-10T09:47:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1019233280 bytes 2021-07-10T09:47:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1062389760 bytes 2021-07-10T09:47:25.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:47:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1087437824 bytes 2021-07-10T09:47:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1116795904 bytes 2021-07-10T09:47:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1117384704 bytes 2021-07-10T09:47:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1195703296 bytes 2021-07-10T09:47:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1230687232 bytes 2021-07-10T09:47:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:47:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:47:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:47:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2021-07-10T09:47:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:47:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_1 2021-07-10T09:47:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_1 2021-07-10T09:47:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2021-07-10T09:47:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a44aa1c6a1 2021-07-10T09:47:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_1 present 11a44aa1c6a1 2021-07-10T09:47:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:47:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a44aa1c6a1 2021-07-10T09:47:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_1 2021-07-10T09:47:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_1 2021-07-10T09:47:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2021-07-10T09:47:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a44aa1c6a1 2021-07-10T09:47:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a44aa1c6a1 2021-07-10T09:47:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a44aa1c6a1 -o 11a44aa1c6a1 = 11a44aa1c6a1 2021-07-10T09:47:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:47:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:47:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_1 2021-07-10T09:47:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2021-07-10T09:47:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:47:28.129+0000 7fe1abc55700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a776b62c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2021-07-10T09:47:28.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:47:28.129+0000 7fe1abc55700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a776b62c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2021-07-10T09:47:32.729 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2021-07-10T09:47:32.730 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:47:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:47:32.731 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2021-07-10T09:47:32.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2021-07-10T09:47:32.731 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2021-07-10T09:47:32.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2021-07-10T09:47:32.735 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2021-07-10T09:47:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:47:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2021-07-10T09:47:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:47:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:47:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_1 2021-07-10T09:47:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_1 2021-07-10T09:47:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2021-07-10T09:47:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:33.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:47:33.167+0000 7f1962f96700 -1 rbd::mirror::ImageMap: 0x55ed0e681320 handle_peer_ack_remove: failed to remove global_image_id=c5232eff-cfe5-4cd9-96bb-1f773587391e 2021-07-10T09:47:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_1: (2) No such file or directory 2021-07-10T09:47:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:47:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:47:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:47:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2021-07-10T09:47:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:47:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:47:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:47:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:47:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:47:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_1 2021-07-10T09:47:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:47:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:47:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_1 2021-07-10T09:47:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2021-07-10T09:47:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:47:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_1: (2) No such file or directory 2021-07-10T09:47:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_1 2021-07-10T09:47:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2021-07-10T09:47:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_1 2021-07-10T09:47:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:47:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:47:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_1 2021-07-10T09:47:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2021-07-10T09:47:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2021-07-10T09:47:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:47:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:47:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:47:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:47:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:47:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_1 2021-07-10T09:47:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2021-07-10T09:47:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2021-07-10T09:47:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:47:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:47:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:47:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:47:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:47:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_1 2021-07-10T09:47:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2021-07-10T09:47:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2021-07-10T09:47:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:47:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:47:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:47:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:47:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:47:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_1 2021-07-10T09:47:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2021-07-10T09:47:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2021-07-10T09:47:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:47:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:47:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:47:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:47:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:47:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_1 2021-07-10T09:47:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:47:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:47:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:47:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2021-07-10T09:47:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2021-07-10T09:47:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:47:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:47:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:47:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:47:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:48:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_1 2021-07-10T09:48:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_1 2021-07-10T09:48:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2021-07-10T09:48:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2021-07-10T09:48:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:48:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_2 2021-07-10T09:48:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_2 512M 2021-07-10T09:48:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:48:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:48:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:48:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_2 2021-07-10T09:48:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_2 2021-07-10T09:48:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:48:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2021-07-10T09:48:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:48:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_2 2021-07-10T09:48:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:48:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:48:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:48:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:48:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 47319040 bytes 2021-07-10T09:48:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 47935488 bytes 2021-07-10T09:48:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 64139264 bytes 2021-07-10T09:48:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 110321664 bytes 2021-07-10T09:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 133541888 bytes 2021-07-10T09:48:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 182262784 bytes 2021-07-10T09:48:07.864 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:48:07.862+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:48:07.864 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:48:07.862+0000 7f25774cc700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:48:07.865 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:48:07.862+0000 7f2578ccf700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:48:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 264031232 bytes 2021-07-10T09:48:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 308894720 bytes 2021-07-10T09:48:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 438557696 bytes 2021-07-10T09:48:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 489875456 bytes 2021-07-10T09:48:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 503157760 bytes 2021-07-10T09:48:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 569669632 bytes 2021-07-10T09:48:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 635938816 bytes 2021-07-10T09:48:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 670567424 bytes 2021-07-10T09:48:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 695657472 bytes 2021-07-10T09:48:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 757487616 bytes 2021-07-10T09:48:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 779709440 bytes 2021-07-10T09:48:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 834415616 bytes 2021-07-10T09:48:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 858399744 bytes 2021-07-10T09:48:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 874876928 bytes 2021-07-10T09:48:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 912216064 bytes 2021-07-10T09:48:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 941763584 bytes 2021-07-10T09:48:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 968111104 bytes 2021-07-10T09:48:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 973672448 bytes 2021-07-10T09:48:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 976671744 bytes 2021-07-10T09:48:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1022864384 bytes 2021-07-10T09:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1118494720 bytes 2021-07-10T09:48:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1162329088 bytes 2021-07-10T09:48:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1282372608 bytes 2021-07-10T09:48:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1298696192 bytes 2021-07-10T09:48:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1389275136 bytes 2021-07-10T09:48:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1395771392 bytes 2021-07-10T09:48:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:48:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:48:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:48:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:48:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2021-07-10T09:48:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:48:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:48:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:48:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:48:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:48:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_2 2021-07-10T09:48:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:48:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:48:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_2 2021-07-10T09:48:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2021-07-10T09:48:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:48:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a46d22a60b 2021-07-10T09:48:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:48:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:48:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_2 present 11a46d22a60b 2021-07-10T09:48:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:48:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:48:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a46d22a60b 2021-07-10T09:48:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:48:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:48:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_2 2021-07-10T09:48:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:48:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:48:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_2 2021-07-10T09:48:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2021-07-10T09:48:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:48:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a46d22a60b 2021-07-10T09:48:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a46d22a60b 2021-07-10T09:48:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a46d22a60b -o 11a46d22a60b = 11a46d22a60b 2021-07-10T09:48:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:48:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:48:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:48:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_2 2021-07-10T09:48:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2021-07-10T09:48:17.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:48:17.146+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a77958fc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2021-07-10T09:48:17.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:48:17.146+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a77958fc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2021-07-10T09:48:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:48:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2021-07-10T09:48:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:48:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:48:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:48:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:48:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:48:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_2 2021-07-10T09:48:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:48:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:48:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_2 2021-07-10T09:48:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2021-07-10T09:48:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:48:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_2: (2) No such file or directory 2021-07-10T09:48:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:48:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:48:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:48:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2021-07-10T09:48:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:48:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:48:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:48:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:48:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:48:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_2 2021-07-10T09:48:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:48:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:48:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_2 2021-07-10T09:48:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2021-07-10T09:48:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:48:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_2: (2) No such file or directory 2021-07-10T09:48:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:48:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:48:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:48:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:48:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_2 2021-07-10T09:48:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2021-07-10T09:48:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_2 2021-07-10T09:48:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:48:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:48:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_2 2021-07-10T09:48:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2021-07-10T09:48:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2021-07-10T09:48:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:48:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:48:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:48:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:48:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:48:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_2 2021-07-10T09:48:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2021-07-10T09:48:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2021-07-10T09:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:48:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:48:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:48:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_2 2021-07-10T09:48:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2021-07-10T09:48:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2021-07-10T09:48:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:48:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:48:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:48:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:48:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:48:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_2 2021-07-10T09:48:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2021-07-10T09:48:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2021-07-10T09:48:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:48:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:48:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:48:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:48:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:48:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_2 2021-07-10T09:48:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2021-07-10T09:48:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2021-07-10T09:48:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:48:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:48:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:48:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:48:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:48:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_2 2021-07-10T09:48:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_2 2021-07-10T09:48:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2021-07-10T09:48:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2021-07-10T09:48:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:48:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:48:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:48:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_3 2021-07-10T09:48:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_3 512M 2021-07-10T09:48:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:48:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:48:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:48:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:48:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:48:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_3 2021-07-10T09:48:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_3 2021-07-10T09:48:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:48:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:48:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2021-07-10T09:48:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:48:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_3 2021-07-10T09:48:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:48:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:48:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=14 2021-07-10T09:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:48:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 68108288 bytes 2021-07-10T09:48:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 85396480 bytes 2021-07-10T09:48:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 108101632 bytes 2021-07-10T09:48:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 155883520 bytes 2021-07-10T09:48:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 183278592 bytes 2021-07-10T09:48:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 229431296 bytes 2021-07-10T09:48:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 270545920 bytes 2021-07-10T09:48:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 295895040 bytes 2021-07-10T09:48:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 341182464 bytes 2021-07-10T09:48:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 393039872 bytes 2021-07-10T09:48:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 418614272 bytes 2021-07-10T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 427517952 bytes 2021-07-10T09:48:59.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 483126272 bytes 2021-07-10T09:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 566678528 bytes 2021-07-10T09:49:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 603724800 bytes 2021-07-10T09:49:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 675961856 bytes 2021-07-10T09:49:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 717376512 bytes 2021-07-10T09:49:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 739636224 bytes 2021-07-10T09:49:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 782671872 bytes 2021-07-10T09:49:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 809323520 bytes 2021-07-10T09:49:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 891320320 bytes 2021-07-10T09:49:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 904859648 bytes 2021-07-10T09:49:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 925483008 bytes 2021-07-10T09:49:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 973089792 bytes 2021-07-10T09:49:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1022460928 bytes 2021-07-10T09:49:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1055494144 bytes 2021-07-10T09:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1104864256 bytes 2021-07-10T09:49:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1172779008 bytes 2021-07-10T09:49:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1188193280 bytes 2021-07-10T09:49:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1193713664 bytes 2021-07-10T09:49:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1246432256 bytes 2021-07-10T09:49:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1331527680 bytes 2021-07-10T09:49:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1396776960 bytes 2021-07-10T09:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1440636928 bytes 2021-07-10T09:49:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1473829888 bytes 2021-07-10T09:49:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1492952064 bytes 2021-07-10T09:49:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3700 IOs, 1504634880 bytes 2021-07-10T09:49:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3800 IOs, 1506444288 bytes 2021-07-10T09:49:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3900 IOs, 1524184064 bytes 2021-07-10T09:49:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4000 IOs, 1548131328 bytes 2021-07-10T09:49:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4100 IOs, 1552471040 bytes 2021-07-10T09:49:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4200 IOs, 1589142528 bytes 2021-07-10T09:49:08.736 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2021-07-10T09:49:08.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2021-07-10T09:49:08.737 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2021-07-10T09:49:08.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:49:08.738+0000 7f1963f98700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 52037) UID: 1000 2021-07-10T09:49:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4300 IOs, 1637956608 bytes 2021-07-10T09:49:09.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:49:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:4400 IOs, 1726284800 bytes 2021-07-10T09:49:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:49:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:49:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:49:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:49:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2021-07-10T09:49:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:49:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:49:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:49:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:49:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_3 2021-07-10T09:49:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_3 2021-07-10T09:49:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2021-07-10T09:49:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a4fb0a4a9 2021-07-10T09:49:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:49:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:49:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_3 present 11a4fb0a4a9 2021-07-10T09:49:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:49:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a4fb0a4a9 2021-07-10T09:49:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:49:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:49:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_3 2021-07-10T09:49:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_3 2021-07-10T09:49:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2021-07-10T09:49:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a4fb0a4a9 2021-07-10T09:49:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a4fb0a4a9 2021-07-10T09:49:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a4fb0a4a9 -o 11a4fb0a4a9 = 11a4fb0a4a9 2021-07-10T09:49:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:49:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:49:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:49:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_3 2021-07-10T09:49:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2021-07-10T09:49:10.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:49:10.189+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a7790dcc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2021-07-10T09:49:10.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:49:10.189+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a7790dcc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2021-07-10T09:49:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:49:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2021-07-10T09:49:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:49:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:49:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:49:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_3 2021-07-10T09:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_3 2021-07-10T09:49:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2021-07-10T09:49:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:15.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi105.stderr:2021-07-10T09:49:15.229+0000 7fe38c0d7700 -1 rbd::mirror::ImageMap: 0x55ab34f42c60 handle_peer_ack_remove: failed to remove global_image_id=91bc3e20-1446-4eb4-8b67-bc81551469e0 2021-07-10T09:49:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_3: (2) No such file or directory 2021-07-10T09:49:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:49:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:49:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:49:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2021-07-10T09:49:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:49:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:49:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:49:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:49:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_3 2021-07-10T09:49:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_3 2021-07-10T09:49:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2021-07-10T09:49:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_3: (2) No such file or directory 2021-07-10T09:49:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:49:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:49:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:49:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:49:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_3 2021-07-10T09:49:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2021-07-10T09:49:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_3 2021-07-10T09:49:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:49:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_3 2021-07-10T09:49:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2021-07-10T09:49:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2021-07-10T09:49:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:49:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:49:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:49:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:49:15.738 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2021-07-10T09:49:15.738 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:49:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:49:15.739 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2021-07-10T09:49:15.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:49:15.740 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:49:15.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:49:15.743 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2021-07-10T09:49:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_3 2021-07-10T09:49:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2021-07-10T09:49:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2021-07-10T09:49:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:49:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:49:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:49:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:49:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_3 2021-07-10T09:49:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2021-07-10T09:49:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2021-07-10T09:49:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:49:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:49:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:49:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:49:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_3 2021-07-10T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2021-07-10T09:49:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2021-07-10T09:49:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:49:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:49:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:49:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:49:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_3 2021-07-10T09:49:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2021-07-10T09:49:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2021-07-10T09:49:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:49:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:49:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:49:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:49:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_3 2021-07-10T09:49:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_3 2021-07-10T09:49:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2021-07-10T09:49:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2021-07-10T09:49:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:49:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:49:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:49:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_4 2021-07-10T09:49:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_4 512M 2021-07-10T09:49:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:49:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:49:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:49:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:49:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:49:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:49:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:49:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_4 2021-07-10T09:49:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_4 2021-07-10T09:49:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:49:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2021-07-10T09:49:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:49:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_4 2021-07-10T09:49:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:49:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:49:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:49:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:49:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 40513536 bytes 2021-07-10T09:49:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 123893760 bytes 2021-07-10T09:49:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 137543680 bytes 2021-07-10T09:49:49.287 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:49:49.285+0000 7f25764ca700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2021-07-10T09:49:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 202537984 bytes 2021-07-10T09:49:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 270058496 bytes 2021-07-10T09:49:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 317038592 bytes 2021-07-10T09:49:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 366608384 bytes 2021-07-10T09:49:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 426236928 bytes 2021-07-10T09:49:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 488624128 bytes 2021-07-10T09:49:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 523610112 bytes 2021-07-10T09:49:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 637612032 bytes 2021-07-10T09:49:52.106 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:49:52.104+0000 7f69eaf37700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:49:52.107 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:49:52.104+0000 7f69e5f2d700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:49:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 696341504 bytes 2021-07-10T09:49:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 714050560 bytes 2021-07-10T09:49:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 714670080 bytes 2021-07-10T09:49:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 756917248 bytes 2021-07-10T09:49:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:49:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:49:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:49:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:49:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2021-07-10T09:49:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:49:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:49:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:49:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:49:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_4 2021-07-10T09:49:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_4 2021-07-10T09:49:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2021-07-10T09:49:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a47426c550 2021-07-10T09:49:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:49:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_4 present 11a47426c550 2021-07-10T09:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:49:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a47426c550 2021-07-10T09:49:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:49:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:49:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_4 2021-07-10T09:49:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_4 2021-07-10T09:49:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2021-07-10T09:49:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a47426c550 2021-07-10T09:49:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a47426c550 2021-07-10T09:49:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a47426c550 -o 11a47426c550 = 11a47426c550 2021-07-10T09:49:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:49:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:49:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:49:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_4 2021-07-10T09:49:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2021-07-10T09:49:54.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:49:54.187+0000 7fe1abc55700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a776b1b800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2021-07-10T09:49:54.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:49:54.187+0000 7fe1abc55700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a776b1b800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2021-07-10T09:49:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:49:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2021-07-10T09:49:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:49:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:49:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:49:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:49:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_4 2021-07-10T09:49:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_4 2021-07-10T09:49:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2021-07-10T09:49:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_4: (2) No such file or directory 2021-07-10T09:49:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:49:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:49:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:49:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2021-07-10T09:49:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:49:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:49:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:49:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:49:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:49:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_4 2021-07-10T09:49:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:49:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:49:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_4 2021-07-10T09:49:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2021-07-10T09:49:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:49:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_4: (2) No such file or directory 2021-07-10T09:49:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:49:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:49:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:49:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:49:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_4 2021-07-10T09:49:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2021-07-10T09:49:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_4 2021-07-10T09:49:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:49:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_4 2021-07-10T09:49:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:49:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:49:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:49:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2021-07-10T09:49:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2021-07-10T09:49:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:49:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:49:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:49:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:49:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:50:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_4 2021-07-10T09:50:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:50:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2021-07-10T09:50:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2021-07-10T09:50:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:50:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_4 2021-07-10T09:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2021-07-10T09:50:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2021-07-10T09:50:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:50:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_4 2021-07-10T09:50:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:50:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2021-07-10T09:50:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2021-07-10T09:50:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:50:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_4 2021-07-10T09:50:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:50:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2021-07-10T09:50:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2021-07-10T09:50:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:50:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_4 2021-07-10T09:50:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_4 2021-07-10T09:50:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2021-07-10T09:50:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2021-07-10T09:50:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:50:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:50:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:50:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_5 2021-07-10T09:50:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_5 512M 2021-07-10T09:50:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:50:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:50:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:50:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:50:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:50:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:50:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:50:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_5 2021-07-10T09:50:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_5 2021-07-10T09:50:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2021-07-10T09:50:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:50:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_5 2021-07-10T09:50:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:50:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=5 2021-07-10T09:50:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:50:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:50:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 60150784 bytes 2021-07-10T09:50:32.785 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:50:32.784+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:50:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 137616384 bytes 2021-07-10T09:50:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 164525056 bytes 2021-07-10T09:50:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 194311168 bytes 2021-07-10T09:50:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 255250432 bytes 2021-07-10T09:50:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 281644032 bytes 2021-07-10T09:50:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 322292736 bytes 2021-07-10T09:50:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 377657344 bytes 2021-07-10T09:50:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 412112896 bytes 2021-07-10T09:50:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 525544448 bytes 2021-07-10T09:50:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 526155776 bytes 2021-07-10T09:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 587060224 bytes 2021-07-10T09:50:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 620004352 bytes 2021-07-10T09:50:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 684812288 bytes 2021-07-10T09:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 710718464 bytes 2021-07-10T09:50:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:50:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:50:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:50:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:50:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2021-07-10T09:50:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:50:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:50:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:50:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:50:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:50:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_5 2021-07-10T09:50:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:50:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:50:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_5 2021-07-10T09:50:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2021-07-10T09:50:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:50:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a4d3b369a3 2021-07-10T09:50:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:50:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:50:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_5 present 11a4d3b369a3 2021-07-10T09:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:50:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a4d3b369a3 2021-07-10T09:50:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:50:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:50:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_5 2021-07-10T09:50:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:50:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:50:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_5 2021-07-10T09:50:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2021-07-10T09:50:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:50:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a4d3b369a3 2021-07-10T09:50:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a4d3b369a3 2021-07-10T09:50:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a4d3b369a3 -o 11a4d3b369a3 = 11a4d3b369a3 2021-07-10T09:50:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:50:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:50:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:50:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_5 2021-07-10T09:50:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2021-07-10T09:50:38.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:50:38.193+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a775e3d800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2021-07-10T09:50:38.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:50:38.193+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a775e3d800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2021-07-10T09:50:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:50:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2021-07-10T09:50:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:50:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:50:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:50:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:50:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:50:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_5 2021-07-10T09:50:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:50:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:50:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_5 2021-07-10T09:50:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2021-07-10T09:50:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:50:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_5: (2) No such file or directory 2021-07-10T09:50:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:50:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:50:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:50:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2021-07-10T09:50:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:50:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:50:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:50:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:50:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:50:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_5 2021-07-10T09:50:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:50:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:50:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_5 2021-07-10T09:50:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2021-07-10T09:50:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:50:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_5: (2) No such file or directory 2021-07-10T09:50:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:50:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:50:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:50:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:50:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_5 2021-07-10T09:50:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2021-07-10T09:50:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_5 2021-07-10T09:50:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:50:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_5 2021-07-10T09:50:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2021-07-10T09:50:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2021-07-10T09:50:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:50:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_5 2021-07-10T09:50:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2021-07-10T09:50:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2021-07-10T09:50:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:50:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_5 2021-07-10T09:50:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2021-07-10T09:50:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2021-07-10T09:50:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:50:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_5 2021-07-10T09:50:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2021-07-10T09:50:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2021-07-10T09:50:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:50:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_5 2021-07-10T09:50:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:50:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:50:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:50:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2021-07-10T09:50:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2021-07-10T09:50:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:50:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:50:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:50:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:50:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:51:11.744 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6124947256786609) > weight (0.25) 2021-07-10T09:51:11.744 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9595479277198591) > weight (0.25) 2021-07-10T09:51:11.745 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9084173318059886) > weight (0.25) 2021-07-10T09:51:11.745 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6251412963040811) > weight (0.25) 2021-07-10T09:51:11.745 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2021-07-10T09:51:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_5 2021-07-10T09:51:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_5 2021-07-10T09:51:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2021-07-10T09:51:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2021-07-10T09:51:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:51:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:51:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:51:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_6 2021-07-10T09:51:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_6 512M 2021-07-10T09:51:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:51:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:51:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:51:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:51:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:51:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:51:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:51:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_6 2021-07-10T09:51:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_6 2021-07-10T09:51:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:51:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2021-07-10T09:51:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:51:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_6 2021-07-10T09:51:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:51:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=10 2021-07-10T09:51:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:51:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:51:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 45571072 bytes 2021-07-10T09:51:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 82937856 bytes 2021-07-10T09:51:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 83549184 bytes 2021-07-10T09:51:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 91176960 bytes 2021-07-10T09:51:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 156313600 bytes 2021-07-10T09:51:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 172446720 bytes 2021-07-10T09:51:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 220481536 bytes 2021-07-10T09:51:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 291316736 bytes 2021-07-10T09:51:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 318675968 bytes 2021-07-10T09:51:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 344743936 bytes 2021-07-10T09:51:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 382957568 bytes 2021-07-10T09:51:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 417107968 bytes 2021-07-10T09:51:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 440530944 bytes 2021-07-10T09:51:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 465013760 bytes 2021-07-10T09:51:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 497110016 bytes 2021-07-10T09:51:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 523496448 bytes 2021-07-10T09:51:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 540040192 bytes 2021-07-10T09:51:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 626666496 bytes 2021-07-10T09:51:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 658118656 bytes 2021-07-10T09:51:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 666156032 bytes 2021-07-10T09:51:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 690616320 bytes 2021-07-10T09:51:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 717188096 bytes 2021-07-10T09:51:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 777667584 bytes 2021-07-10T09:51:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 806565888 bytes 2021-07-10T09:51:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 869039104 bytes 2021-07-10T09:51:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 932586496 bytes 2021-07-10T09:51:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 984980480 bytes 2021-07-10T09:51:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1055077376 bytes 2021-07-10T09:51:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1093600256 bytes 2021-07-10T09:51:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1130986496 bytes 2021-07-10T09:51:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1188531200 bytes 2021-07-10T09:51:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1279874048 bytes 2021-07-10T09:51:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:51:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:51:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:51:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:51:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2021-07-10T09:51:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:51:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:51:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:51:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:51:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:51:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_6 2021-07-10T09:51:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:51:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:51:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_6 2021-07-10T09:51:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2021-07-10T09:51:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:51:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a46f121863 2021-07-10T09:51:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:51:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:51:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_6 present 11a46f121863 2021-07-10T09:51:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:51:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:51:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a46f121863 2021-07-10T09:51:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:51:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:51:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_6 2021-07-10T09:51:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:51:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:51:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_6 2021-07-10T09:51:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2021-07-10T09:51:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:51:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a46f121863 2021-07-10T09:51:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a46f121863 2021-07-10T09:51:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a46f121863 -o 11a46f121863 = 11a46f121863 2021-07-10T09:51:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:51:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:51:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:51:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_6 2021-07-10T09:51:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2021-07-10T09:51:26.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:51:26.213+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a7790dcc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2021-07-10T09:51:26.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:51:26.213+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a7790dcc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2021-07-10T09:51:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:51:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2021-07-10T09:51:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:51:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:51:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:51:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:51:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:51:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_6 2021-07-10T09:51:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:51:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:51:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_6 2021-07-10T09:51:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2021-07-10T09:51:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:51:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_6: (2) No such file or directory 2021-07-10T09:51:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:51:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:51:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:51:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2021-07-10T09:51:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:51:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:51:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:51:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:51:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:51:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_6 2021-07-10T09:51:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:51:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:51:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_6 2021-07-10T09:51:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2021-07-10T09:51:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:51:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_6: (2) No such file or directory 2021-07-10T09:51:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:51:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:51:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:51:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:51:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_6 2021-07-10T09:51:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2021-07-10T09:51:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_6 2021-07-10T09:51:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:51:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:51:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_6 2021-07-10T09:51:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2021-07-10T09:51:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2021-07-10T09:51:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:51:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:51:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:51:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:51:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:51:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_6 2021-07-10T09:51:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2021-07-10T09:51:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2021-07-10T09:51:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:51:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:51:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:51:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:51:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:51:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_6 2021-07-10T09:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2021-07-10T09:51:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2021-07-10T09:51:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:51:34.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:51:34.775 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:51:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:51:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:51:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_6 2021-07-10T09:51:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2021-07-10T09:51:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2021-07-10T09:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:51:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_6 2021-07-10T09:51:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:51:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:51:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:51:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2021-07-10T09:51:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2021-07-10T09:51:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:51:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:51:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:51:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:51:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:52:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_6 2021-07-10T09:52:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_6 2021-07-10T09:52:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2021-07-10T09:52:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2021-07-10T09:52:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:52:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:52:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:52:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_7 2021-07-10T09:52:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_7 512M 2021-07-10T09:52:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:52:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:52:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:52:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_7 2021-07-10T09:52:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_7 2021-07-10T09:52:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:52:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2021-07-10T09:52:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:52:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_7 2021-07-10T09:52:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:52:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=8 2021-07-10T09:52:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:52:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:52:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 33439744 bytes 2021-07-10T09:52:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 62513152 bytes 2021-07-10T09:52:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 87470080 bytes 2021-07-10T09:52:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 145313792 bytes 2021-07-10T09:52:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 145941504 bytes 2021-07-10T09:52:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 166670336 bytes 2021-07-10T09:52:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 228415488 bytes 2021-07-10T09:52:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 291986432 bytes 2021-07-10T09:52:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 331591680 bytes 2021-07-10T09:52:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 386382848 bytes 2021-07-10T09:52:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 419653632 bytes 2021-07-10T09:52:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 423890944 bytes 2021-07-10T09:52:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 449423360 bytes 2021-07-10T09:52:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 481661952 bytes 2021-07-10T09:52:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 528491520 bytes 2021-07-10T09:52:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 559248384 bytes 2021-07-10T09:52:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 618984448 bytes 2021-07-10T09:52:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 639636480 bytes 2021-07-10T09:52:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 671269888 bytes 2021-07-10T09:52:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 700526592 bytes 2021-07-10T09:52:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 745613312 bytes 2021-07-10T09:52:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 826118144 bytes 2021-07-10T09:52:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 826741760 bytes 2021-07-10T09:52:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 920193024 bytes 2021-07-10T09:52:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 931849216 bytes 2021-07-10T09:52:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:52:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:52:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:52:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:52:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2021-07-10T09:52:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:52:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:52:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:52:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:52:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:52:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_7 2021-07-10T09:52:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:52:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:52:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_7 2021-07-10T09:52:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2021-07-10T09:52:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:52:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a41b4e2967 2021-07-10T09:52:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:52:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:52:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_7 present 11a41b4e2967 2021-07-10T09:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a41b4e2967 2021-07-10T09:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_7 2021-07-10T09:52:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:52:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:52:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_7 2021-07-10T09:52:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2021-07-10T09:52:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:52:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a41b4e2967 2021-07-10T09:52:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a41b4e2967 2021-07-10T09:52:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a41b4e2967 -o 11a41b4e2967 = 11a41b4e2967 2021-07-10T09:52:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:52:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:52:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:52:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_7 2021-07-10T09:52:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2021-07-10T09:52:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:52:13.240+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a776b6ac00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2021-07-10T09:52:13.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:52:13.240+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a776b6ac00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2021-07-10T09:52:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:52:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2021-07-10T09:52:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:52:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:52:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:52:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:52:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:52:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_7 2021-07-10T09:52:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:52:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:52:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_7 2021-07-10T09:52:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2021-07-10T09:52:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:52:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_7: (2) No such file or directory 2021-07-10T09:52:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:52:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:52:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:52:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2021-07-10T09:52:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:52:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:52:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:52:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:52:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:52:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_7 2021-07-10T09:52:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:52:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:52:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_7 2021-07-10T09:52:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2021-07-10T09:52:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:52:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_7: (2) No such file or directory 2021-07-10T09:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_7 2021-07-10T09:52:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2021-07-10T09:52:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_7 2021-07-10T09:52:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:52:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_7 2021-07-10T09:52:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2021-07-10T09:52:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2021-07-10T09:52:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:52:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:52:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:52:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:52:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_7 2021-07-10T09:52:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2021-07-10T09:52:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2021-07-10T09:52:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:52:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:52:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:52:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:52:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:52:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_7 2021-07-10T09:52:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2021-07-10T09:52:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2021-07-10T09:52:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:52:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:52:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:52:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:52:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:52:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_7 2021-07-10T09:52:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2021-07-10T09:52:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2021-07-10T09:52:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:52:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:52:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:52:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:52:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:52:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_7 2021-07-10T09:52:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2021-07-10T09:52:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2021-07-10T09:52:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:52:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:52:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:52:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:52:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:52:34.746 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2021-07-10T09:52:34.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2021-07-10T09:52:34.747 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2021-07-10T09:52:34.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:2021-07-10T09:52:34.747+0000 7f5866a8d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 72845) UID: 1000 2021-07-10T09:52:34.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi105.stderr:daemon-helper: command failed with exit status 1 2021-07-10T09:52:35.748 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2021-07-10T09:52:35.748 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:52:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-10T09:52:35.749 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2021-07-10T09:52:35.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2021-07-10T09:52:35.750 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2021-07-10T09:52:35.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2021-07-10T09:52:35.754 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2021-07-10T09:52:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_7 2021-07-10T09:52:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_7 2021-07-10T09:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2021-07-10T09:52:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2021-07-10T09:52:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:52:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:52:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:52:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_8 2021-07-10T09:52:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_8 512M 2021-07-10T09:52:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:52:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:52:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:52:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:52:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:52:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_8 2021-07-10T09:52:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_8 2021-07-10T09:52:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:52:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:52:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2021-07-10T09:52:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:52:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_8 2021-07-10T09:52:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:52:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:52:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:52:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=12 2021-07-10T09:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:52:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:52:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 47419392 bytes 2021-07-10T09:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 90104832 bytes 2021-07-10T09:52:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 119115776 bytes 2021-07-10T09:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 141925376 bytes 2021-07-10T09:52:52.967 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:52:52.965+0000 7f5116a99700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 5 2021-07-10T09:52:52.967 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T09:52:52.966+0000 7f510fa8b700 -1 bdev(0x55a8ca082000 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2021-07-10T09:52:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 313231360 bytes 2021-07-10T09:52:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 313871360 bytes 2021-07-10T09:52:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 324084736 bytes 2021-07-10T09:52:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 349778944 bytes 2021-07-10T09:52:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 388840448 bytes 2021-07-10T09:52:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 436877312 bytes 2021-07-10T09:52:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 465864704 bytes 2021-07-10T09:52:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 518515712 bytes 2021-07-10T09:52:55.330 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:52:55.328+0000 7f25744c6700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:52:55.331 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:52:55.328+0000 7f25724c2700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:52:55.331 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:52:55.328+0000 7f2575cc9700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:52:55.331 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:52:55.329+0000 7f2576ccb700 -1 bdev(0x5556dc256000 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2021-07-10T09:52:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 602253312 bytes 2021-07-10T09:52:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 620440576 bytes 2021-07-10T09:52:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 635863040 bytes 2021-07-10T09:52:56.287 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T09:52:56.285+0000 7f69e8732700 -1 bdev(0x56305ebfc000 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2021-07-10T09:52:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 675257344 bytes 2021-07-10T09:52:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 725740544 bytes 2021-07-10T09:52:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 739708928 bytes 2021-07-10T09:52:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 795145216 bytes 2021-07-10T09:52:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 825079808 bytes 2021-07-10T09:52:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2100 IOs, 864478208 bytes 2021-07-10T09:52:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2200 IOs, 865083392 bytes 2021-07-10T09:52:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2300 IOs, 923637760 bytes 2021-07-10T09:52:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2400 IOs, 985065472 bytes 2021-07-10T09:52:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2500 IOs, 1009500160 bytes 2021-07-10T09:52:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2600 IOs, 1061676032 bytes 2021-07-10T09:52:59.741 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T09:52:59.740+0000 7f25844e6700 -1 reset not still connected to 0x55571d2c60d0 2021-07-10T09:52:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2700 IOs, 1091109888 bytes 2021-07-10T09:53:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2800 IOs, 1114209280 bytes 2021-07-10T09:53:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2900 IOs, 1126037504 bytes 2021-07-10T09:53:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3000 IOs, 1177140224 bytes 2021-07-10T09:53:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3100 IOs, 1266648064 bytes 2021-07-10T09:53:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3200 IOs, 1288958976 bytes 2021-07-10T09:53:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3300 IOs, 1382197248 bytes 2021-07-10T09:53:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3400 IOs, 1424089088 bytes 2021-07-10T09:53:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3500 IOs, 1466512384 bytes 2021-07-10T09:53:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:3600 IOs, 1493684224 bytes 2021-07-10T09:53:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:53:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:53:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:53:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:53:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2021-07-10T09:53:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:53:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:53:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:53:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_8 2021-07-10T09:53:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:53:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:53:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_8 2021-07-10T09:53:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2021-07-10T09:53:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:53:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id=11a4ccde40a7 2021-07-10T09:53:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:53:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:53:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_8 present 11a4ccde40a7 2021-07-10T09:53:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:53:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T09:53:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id=11a4ccde40a7 2021-07-10T09:53:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:53:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:53:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_8 2021-07-10T09:53:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:53:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:53:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_8 2021-07-10T09:53:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2021-07-10T09:53:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:53:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id=11a4ccde40a7 2021-07-10T09:53:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n 11a4ccde40a7 2021-07-10T09:53:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -z 11a4ccde40a7 -o 11a4ccde40a7 = 11a4ccde40a7 2021-07-10T09:53:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_state=present 2021-07-10T09:53:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = present 2021-07-10T09:53:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:53:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ disable_mirror cluster2 mirror image_8 2021-07-10T09:53:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2021-07-10T09:53:04.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:53:04.250+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a77910c800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2021-07-10T09:53:04.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi105.stderr:2021-07-10T09:53:04.250+0000 7fe1ab454700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a77910c800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=28254c44-de8b-4d4c-bece-57c7d09a96b4, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2021-07-10T09:53:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring disabled 2021-07-10T09:53:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2021-07-10T09:53:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:53:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=deleted 2021-07-10T09:53:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:53:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:53:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:53:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_8 2021-07-10T09:53:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:53:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:53:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_8 2021-07-10T09:53:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2021-07-10T09:53:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:53:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_8: (2) No such file or directory 2021-07-10T09:53:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T09:53:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T09:53:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T09:53:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2021-07-10T09:53:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:53:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=deleted 2021-07-10T09:53:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:53:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T09:53:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T09:53:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_8 2021-07-10T09:53:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:53:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:53:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_8 2021-07-10T09:53:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2021-07-10T09:53:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T09:53:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error opening image image_8: (2) No such file or directory 2021-07-10T09:53:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T09:53:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:53:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test deleted = deleted 2021-07-10T09:53:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:53:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ purge_snapshots cluster2 mirror image_8 2021-07-10T09:53:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2021-07-10T09:53:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image_retry cluster2 mirror image_8 2021-07-10T09:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:53:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0 2021-07-10T09:53:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_8 2021-07-10T09:53:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2021-07-10T09:53:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2021-07-10T09:53:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:53:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:53:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:53:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:53:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T09:53:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_8 2021-07-10T09:53:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2021-07-10T09:53:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2021-07-10T09:53:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:53:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:53:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:53:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:53:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T09:53:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_8 2021-07-10T09:53:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2021-07-10T09:53:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2021-07-10T09:53:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:53:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:53:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:53:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:53:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T09:53:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_8 2021-07-10T09:53:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2021-07-10T09:53:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2021-07-10T09:53:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:53:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:53:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:53:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:53:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T09:53:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_8 2021-07-10T09:53:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2021-07-10T09:53:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2021-07-10T09:53:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 0% complete...failed. 2021-07-10T09:53:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: error: image still has watchers 2021-07-10T09:53:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2021-07-10T09:53:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0 1 2 4 8 16 32 2021-07-10T09:53:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 16 2021-07-10T09:53:27.074 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.2 2021-07-10T09:53:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 6 2021-07-10T09:53:27.075 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2021-07-10T09:53:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:*** Caught signal (Aborted) ** 2021-07-10T09:53:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: in thread 7f29ecb57540 thread_name:rbd-mirror 2021-07-10T09:53:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: ceph version 17.0.0-5861-g92adad34 (92adad3411b7280fb159eb9d7304c654695beb8b) quincy (dev) 2021-07-10T09:53:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f29ea3c0b20] 2021-07-10T09:53:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 2: pthread_cond_wait() 2021-07-10T09:53:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f29e91cb8f0] 2021-07-10T09:53:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 4: (C_SaferCond::wait()+0x63) [0x5629da4c7353] 2021-07-10T09:53:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x5629da4caf43] 2021-07-10T09:53:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5629da4cb152] 2021-07-10T09:53:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5629da4c2a71] 2021-07-10T09:53:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 8: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5629da4bcae5] 2021-07-10T09:53:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 9: main() 2021-07-10T09:53:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 10: __libc_start_main() 2021-07-10T09:53:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 11: _start() 2021-07-10T09:53:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:2021-07-10T09:53:27.076+0000 7f29ecb57540 -1 *** Caught signal (Aborted) ** 2021-07-10T09:53:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: in thread 7f29ecb57540 thread_name:rbd-mirror 2021-07-10T09:53:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 2021-07-10T09:53:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: ceph version 17.0.0-5861-g92adad34 (92adad3411b7280fb159eb9d7304c654695beb8b) quincy (dev) 2021-07-10T09:53:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f29ea3c0b20] 2021-07-10T09:53:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 2: pthread_cond_wait() 2021-07-10T09:53:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f29e91cb8f0] 2021-07-10T09:53:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 4: (C_SaferCond::wait()+0x63) [0x5629da4c7353] 2021-07-10T09:53:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x5629da4caf43] 2021-07-10T09:53:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5629da4cb152] 2021-07-10T09:53:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5629da4c2a71] 2021-07-10T09:53:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 8: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5629da4bcae5] 2021-07-10T09:53:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 9: main() 2021-07-10T09:53:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 10: __libc_start_main() 2021-07-10T09:53:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 11: _start() 2021-07-10T09:53:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-07-10T09:53:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 2021-07-10T09:53:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: -1819> 2021-07-10T09:43:18.922+0000 7f29e388a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 57637) UID: 1000 2021-07-10T09:53:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 0> 2021-07-10T09:53:27.076+0000 7f29ecb57540 -1 *** Caught signal (Aborted) ** 2021-07-10T09:53:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: in thread 7f29ecb57540 thread_name:rbd-mirror 2021-07-10T09:53:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 2021-07-10T09:53:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: ceph version 17.0.0-5861-g92adad34 (92adad3411b7280fb159eb9d7304c654695beb8b) quincy (dev) 2021-07-10T09:53:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f29ea3c0b20] 2021-07-10T09:53:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 2: pthread_cond_wait() 2021-07-10T09:53:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f29e91cb8f0] 2021-07-10T09:53:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 4: (C_SaferCond::wait()+0x63) [0x5629da4c7353] 2021-07-10T09:53:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x5629da4caf43] 2021-07-10T09:53:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5629da4cb152] 2021-07-10T09:53:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5629da4c2a71] 2021-07-10T09:53:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 8: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5629da4bcae5] 2021-07-10T09:53:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 9: main() 2021-07-10T09:53:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 10: __libc_start_main() 2021-07-10T09:53:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 11: _start() 2021-07-10T09:53:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-07-10T09:53:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr: 2021-07-10T09:53:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi105.stderr:daemon-helper: command crashed with signal 11 2021-07-10T09:53:29.210 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2021-07-10T09:53:29.210 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-07-10T09:53:29.714 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2021-07-10T09:53:29.714 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2021-07-10T09:53:29.715 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2021-07-10T09:53:29.715 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2021-07-10T09:53:29.716 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2021-07-10T09:53:29.716 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:53:29.714+0000 7f3197905700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 31722) UID: 0 2021-07-10T09:53:29.717 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:53:29.714+0000 7f3197905700 -1 osd.0 444 *** Got signal Terminated *** 2021-07-10T09:53:29.717 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2021-07-10T09:53:29.714+0000 7f3197905700 -1 osd.0 444 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-07-10T09:53:29.718 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:53:29.714+0000 7f72e9934700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 31725) UID: 0 2021-07-10T09:53:29.718 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:53:29.714+0000 7f72e9934700 -1 osd.1 444 *** Got signal Terminated *** 2021-07-10T09:53:29.718 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2021-07-10T09:53:29.714+0000 7f72e9934700 -1 osd.1 444 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-07-10T09:53:29.719 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:53:29.715+0000 7f4248214700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 31728) UID: 0 2021-07-10T09:53:29.719 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:53:29.715+0000 7f4248214700 -1 osd.2 444 *** Got signal Terminated *** 2021-07-10T09:53:29.720 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2021-07-10T09:53:29.715+0000 7f4248214700 -1 osd.2 444 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-07-10T09:53:29.720 INFO:tasks.ceph.cluster1.mon.a.smithi062.stderr:2021-07-10T09:53:29.715+0000 7f4c7dd6d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 31378) UID: 0 2021-07-10T09:53:29.720 INFO:tasks.ceph.cluster1.mon.a.smithi062.stderr:2021-07-10T09:53:29.715+0000 7f4c7dd6d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-07-10T09:53:29.721 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2021-07-10T09:53:29.715+0000 7fe422649700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 31381) UID: 0 2021-07-10T09:53:29.721 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2021-07-10T09:53:29.715+0000 7fe422649700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2021-07-10T09:53:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ remove_image cluster2 mirror image_8 2021-07-10T09:53:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_8 2021-07-10T09:53:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2021-07-10T09:53:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2021-07-10T09:53:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2021-07-10T09:53:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:53:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for i in `seq 1 10` 2021-07-10T09:53:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image=image_9 2021-07-10T09:53:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ create_image cluster2 mirror image_9 512M 2021-07-10T09:53:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:53:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:53:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:53:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:53:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local size=128 2021-07-10T09:53:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' -n 512M ']' 2021-07-10T09:53:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ size=512M 2021-07-10T09:53:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ shift 2021-07-10T09:53:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_9 2021-07-10T09:53:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ enable_mirror cluster2 mirror image_9 2021-07-10T09:53:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:53:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local mode=snapshot 2021-07-10T09:53:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2021-07-10T09:53:45.755 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9233617912970337) > weight (0.25) 2021-07-10T09:53:45.755 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5289982135047041) > weight (0.25) 2021-07-10T09:53:45.756 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9038121116845743) > weight (0.25) 2021-07-10T09:53:45.756 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2021-07-10T09:53:45.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2021-07-10T09:53:45.757 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2021-07-10T09:53:45.757 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2021-07-10T09:53:45.757 INFO:teuthology.orchestra.run:waiting for 600 2021-07-10T09:53:45.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:2021-07-10T09:53:45.756+0000 7fde99033700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 62925) UID: 1000 2021-07-10T09:53:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:Mirroring enabled 2021-07-10T09:53:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ stress_write_image cluster2 mirror image_9 2021-07-10T09:53:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster2 2021-07-10T09:53:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:53:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2021-07-10T09:53:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local duration=7 2021-07-10T09:53:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set +e 2021-07-10T09:53:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2021-07-10T09:53:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:100 IOs, 24112128 bytes 2021-07-10T09:53:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:200 IOs, 85800960 bytes 2021-07-10T09:53:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:300 IOs, 130582528 bytes 2021-07-10T09:53:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:400 IOs, 165784576 bytes 2021-07-10T09:53:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:500 IOs, 214026240 bytes 2021-07-10T09:53:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:600 IOs, 238745600 bytes 2021-07-10T09:53:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:700 IOs, 319927296 bytes 2021-07-10T09:53:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:800 IOs, 387948544 bytes 2021-07-10T09:53:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:900 IOs, 517514240 bytes 2021-07-10T09:53:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1000 IOs, 537882624 bytes 2021-07-10T09:53:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1100 IOs, 557294592 bytes 2021-07-10T09:53:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1200 IOs, 597156864 bytes 2021-07-10T09:53:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1300 IOs, 616368128 bytes 2021-07-10T09:53:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1400 IOs, 713039872 bytes 2021-07-10T09:53:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1500 IOs, 727748608 bytes 2021-07-10T09:53:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1600 IOs, 769350656 bytes 2021-07-10T09:53:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1700 IOs, 836628480 bytes 2021-07-10T09:53:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1800 IOs, 871055360 bytes 2021-07-10T09:53:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:1900 IOs, 998305792 bytes 2021-07-10T09:53:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:2000 IOs, 1012092928 bytes 2021-07-10T09:53:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ error_code=124 2021-07-10T09:53:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ set -e 2021-07-10T09:53:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ '[' 124 -eq 124 ']' 2021-07-10T09:53:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ return 0 2021-07-10T09:53:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2021-07-10T09:53:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T09:53:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T09:53:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T09:53:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local state=present 2021-07-10T09:53:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T09:53:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local s 2021-07-10T09:53:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T09:53:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T09:53:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T09:53:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T09:53:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T09:53:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T09:53:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T10:03:49.920 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.3 2021-07-10T10:03:49.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 6 2021-07-10T10:03:49.924 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_92adad3411b7280fb159eb9d7304c654695beb8b/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_24e3280d7d1fc156628af95b24b27a29e38b2766/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2021-07-10T10:03:49.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:*** Caught signal (Aborted) ** 2021-07-10T10:03:49.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: in thread 7fdea2300540 thread_name:rbd-mirror 2021-07-10T10:03:49.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: ceph version 17.0.0-5861-g92adad34 (92adad3411b7280fb159eb9d7304c654695beb8b) quincy (dev) 2021-07-10T10:03:49.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fde9fb69b20] 2021-07-10T10:03:49.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2: pthread_cond_wait() 2021-07-10T10:03:49.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fde9e9748f0] 2021-07-10T10:03:49.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 4: (C_SaferCond::wait()+0x63) [0x55fb702aa353] 2021-07-10T10:03:49.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x55fb702adf43] 2021-07-10T10:03:49.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x55fb702ae152] 2021-07-10T10:03:49.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x55fb702a5a71] 2021-07-10T10:03:49.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x55fb702a5a59] 2021-07-10T10:03:49.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x55fb7029fae5] 2021-07-10T10:03:49.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 10: main() 2021-07-10T10:03:49.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 11: __libc_start_main() 2021-07-10T10:03:49.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 12: _start() 2021-07-10T10:03:49.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:2021-07-10T10:03:49.925+0000 7fdea2300540 -1 *** Caught signal (Aborted) ** 2021-07-10T10:03:49.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: in thread 7fdea2300540 thread_name:rbd-mirror 2021-07-10T10:03:49.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2021-07-10T10:03:49.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: ceph version 17.0.0-5861-g92adad34 (92adad3411b7280fb159eb9d7304c654695beb8b) quincy (dev) 2021-07-10T10:03:49.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fde9fb69b20] 2021-07-10T10:03:49.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2: pthread_cond_wait() 2021-07-10T10:03:49.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fde9e9748f0] 2021-07-10T10:03:49.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 4: (C_SaferCond::wait()+0x63) [0x55fb702aa353] 2021-07-10T10:03:49.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x55fb702adf43] 2021-07-10T10:03:49.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x55fb702ae152] 2021-07-10T10:03:49.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x55fb702a5a71] 2021-07-10T10:03:49.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x55fb702a5a59] 2021-07-10T10:03:49.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x55fb7029fae5] 2021-07-10T10:03:49.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 10: main() 2021-07-10T10:03:49.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 11: __libc_start_main() 2021-07-10T10:03:49.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 12: _start() 2021-07-10T10:03:49.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-07-10T10:03:49.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2021-07-10T10:03:49.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: -2145> 2021-07-10T09:53:45.756+0000 7fde99033700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 62925) UID: 1000 2021-07-10T10:03:49.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 0> 2021-07-10T10:03:49.925+0000 7fdea2300540 -1 *** Caught signal (Aborted) ** 2021-07-10T10:03:49.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: in thread 7fdea2300540 thread_name:rbd-mirror 2021-07-10T10:03:49.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2021-07-10T10:03:49.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: ceph version 17.0.0-5861-g92adad34 (92adad3411b7280fb159eb9d7304c654695beb8b) quincy (dev) 2021-07-10T10:03:49.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fde9fb69b20] 2021-07-10T10:03:49.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2: pthread_cond_wait() 2021-07-10T10:03:49.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fde9e9748f0] 2021-07-10T10:03:49.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 4: (C_SaferCond::wait()+0x63) [0x55fb702aa353] 2021-07-10T10:03:49.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x55fb702adf43] 2021-07-10T10:03:49.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x55fb702ae152] 2021-07-10T10:03:49.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x55fb702a5a71] 2021-07-10T10:03:49.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x55fb702a5a59] 2021-07-10T10:03:49.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x55fb7029fae5] 2021-07-10T10:03:49.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 10: main() 2021-07-10T10:03:49.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 11: __libc_start_main() 2021-07-10T10:03:49.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 12: _start() 2021-07-10T10:03:49.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-07-10T10:03:49.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2021-07-10T10:03:49.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: -9999> 2021-07-10T09:53:45.756+0000 7fde99033700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 62925) UID: 1000 2021-07-10T10:03:49.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: -9998> 2021-07-10T10:03:49.925+0000 7fdea2300540 -1 *** Caught signal (Aborted) ** 2021-07-10T10:03:49.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: in thread 7fdea2300540 thread_name:rbd-mirror 2021-07-10T10:03:49.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2021-07-10T10:03:49.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: ceph version 17.0.0-5861-g92adad34 (92adad3411b7280fb159eb9d7304c654695beb8b) quincy (dev) 2021-07-10T10:03:49.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fde9fb69b20] 2021-07-10T10:03:49.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2: pthread_cond_wait() 2021-07-10T10:03:49.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fde9e9748f0] 2021-07-10T10:03:49.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 4: (C_SaferCond::wait()+0x63) [0x55fb702aa353] 2021-07-10T10:03:49.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x55fb702adf43] 2021-07-10T10:03:49.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x55fb702ae152] 2021-07-10T10:03:49.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x55fb702a5a71] 2021-07-10T10:03:49.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x55fb702a5a59] 2021-07-10T10:03:49.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x55fb7029fae5] 2021-07-10T10:03:49.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 10: main() 2021-07-10T10:03:49.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 11: __libc_start_main() 2021-07-10T10:03:49.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 12: _start() 2021-07-10T10:03:49.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-07-10T10:03:49.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr: 2021-07-10T10:03:50.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi105.stderr:daemon-helper: command crashed with signal 6 2021-07-10T10:03:51.254 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster2] failed 2021-07-10T10:03:51.254 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-07-10T10:03:51.757 INFO:tasks.ceph.cluster2.osd.0:Sent signal 15 2021-07-10T10:03:51.758 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2021-07-10T10:03:51.758 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2021-07-10T10:03:51.758 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2021-07-10T10:03:51.759 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2021-07-10T10:03:51.760 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T10:03:51.757+0000 7f2596568700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 (PID: 32326) UID: 0 2021-07-10T10:03:51.760 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T10:03:51.757+0000 7f2596568700 -1 osd.0 443 *** Got signal Terminated *** 2021-07-10T10:03:51.760 INFO:tasks.ceph.cluster2.osd.0.smithi105.stderr:2021-07-10T10:03:51.757+0000 7f2596568700 -1 osd.0 443 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-07-10T10:03:51.761 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T10:03:51.757+0000 7f5134332700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 32328) UID: 0 2021-07-10T10:03:51.761 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T10:03:51.757+0000 7f5134332700 -1 osd.1 443 *** Got signal Terminated *** 2021-07-10T10:03:51.762 INFO:tasks.ceph.cluster2.osd.1.smithi105.stderr:2021-07-10T10:03:51.757+0000 7f5134332700 -1 osd.1 443 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-07-10T10:03:51.763 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T10:03:51.758+0000 7f6a087d0700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 32331) UID: 0 2021-07-10T10:03:51.763 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T10:03:51.758+0000 7f6a087d0700 -1 osd.2 443 *** Got signal Terminated *** 2021-07-10T10:03:51.763 INFO:tasks.ceph.cluster2.osd.2.smithi105.stderr:2021-07-10T10:03:51.758+0000 7f6a087d0700 -1 osd.2 443 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-07-10T10:03:51.764 INFO:tasks.ceph.cluster2.mon.a.smithi105.stderr:2021-07-10T10:03:51.758+0000 7f4d943c1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 32020) UID: 0 2021-07-10T10:03:51.764 INFO:tasks.ceph.cluster2.mon.a.smithi105.stderr:2021-07-10T10:03:51.758+0000 7f4d943c1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-07-10T10:03:51.765 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2021-07-10T10:03:51.759+0000 7fc52f50e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 35856) UID: 0 2021-07-10T10:03:51.765 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2021-07-10T10:03:51.759+0000 7fc52f50e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2021-07-10T10:43:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T10:43:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ image_id= 2021-07-10T10:43:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T10:43:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 0.1 2021-07-10T10:43:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_9 present '' 2021-07-10T10:43:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T10:43:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T10:43:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T10:43:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T10:43:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T10:43:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T10:43:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T10:43:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T10:43:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T10:43:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T10:43:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T10:43:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T10:43:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T11:33:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T11:33:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T11:33:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T11:33:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = deleted 2021-07-10T11:33:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T11:33:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 1 2021-07-10T11:33:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_9 present '' 2021-07-10T11:33:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T11:33:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T11:33:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T11:33:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T11:33:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T11:33:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T11:33:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T11:33:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T11:33:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T11:33:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T11:33:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T11:33:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T11:33:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T12:23:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T12:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T12:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T12:23:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = deleted 2021-07-10T12:23:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T12:23:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 2 2021-07-10T12:23:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_9 present '' 2021-07-10T12:23:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T12:23:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T12:23:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T12:23:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T12:23:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T12:23:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T12:23:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T12:23:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T12:23:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T12:23:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T12:23:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T12:23:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T12:23:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T13:13:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T13:13:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T13:13:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T13:13:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = deleted 2021-07-10T13:13:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T13:13:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 4 2021-07-10T13:14:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_9 present '' 2021-07-10T13:14:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T13:14:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T13:14:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T13:14:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T13:14:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T13:14:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T13:14:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T13:14:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T13:14:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T13:14:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T13:14:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T13:14:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T13:14:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T14:04:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T14:04:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T14:04:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T14:04:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = deleted 2021-07-10T14:04:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T14:04:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T14:04:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_9 present '' 2021-07-10T14:04:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T14:04:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T14:04:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T14:04:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T14:04:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T14:04:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T14:04:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T14:04:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T14:04:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T14:04:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T14:04:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T14:04:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T14:04:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T14:54:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T14:54:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T14:54:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test -n '' 2021-07-10T14:54:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test present = deleted 2021-07-10T14:54:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2021-07-10T14:54:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ sleep 8 2021-07-10T14:54:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ test_image_present cluster1 mirror image_9 present '' 2021-07-10T14:54:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local cluster=cluster1 2021-07-10T14:54:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local pool=mirror 2021-07-10T14:54:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image=image_9 2021-07-10T14:54:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local test_state=present 2021-07-10T14:54:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local image_id= 2021-07-10T14:54:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_state=deleted 2021-07-10T14:54:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ local current_image_id 2021-07-10T14:54:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ get_image_id cluster1 mirror image_9 2021-07-10T14:54:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster=cluster1 2021-07-10T14:54:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local pool=mirror 2021-07-10T14:54:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local image=image_9 2021-07-10T14:54:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2021-07-10T14:54:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2021-07-10T15:20:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+ current_image_id= 2021-07-10T15:20:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cluster1 status 2021-07-10T15:20:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ cleanup 143 2021-07-10T15:20:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local error_code=143 2021-07-10T15:20:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ set +e 2021-07-10T15:20:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ '[' 143 -ne 0 ']' 2021-07-10T15:20:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ status 2021-07-10T15:20:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ local cluster daemon image_pool image_ns image 2021-07-10T15:20:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for cluster in ${CLUSTER1} ${CLUSTER2} 2021-07-10T15:20:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 'cluster1 status' 2021-07-10T15:20:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_ARGS= 2021-07-10T15:20:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --cluster cluster1 -s 2021-07-10T15:25:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T15:25:06.338+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T15:30:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T15:30:06.340+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T15:35:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T15:35:06.340+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T15:40:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T15:40:06.341+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T15:45:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T15:45:06.341+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T15:50:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T15:50:06.342+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T15:55:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T15:55:06.342+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:00:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:00:06.343+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:05:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:05:06.344+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:10:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:10:06.344+0000 7f6348500700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:10:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2021-07-10T16:10:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_ARGS= 2021-07-10T16:10:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --cluster cluster1 service dump 2021-07-10T16:15:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:15:06.447+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:20:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:20:06.448+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:25:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:25:06.448+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:30:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:30:06.449+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:35:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:35:06.449+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:40:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:40:06.450+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:45:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:45:06.450+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:50:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:50:06.450+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T16:55:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T16:55:06.451+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:00:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:00:06.451+0000 7f8aa6b2b700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:00:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2021-07-10T17:00:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_ARGS= 2021-07-10T17:00:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ ceph --cluster cluster1 service status 2021-07-10T17:05:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:05:06.554+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:10:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:10:06.555+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:15:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:15:06.555+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:20:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:20:06.556+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:25:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:25:06.557+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:30:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:30:06.557+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:35:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:35:06.558+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:40:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:40:06.557+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:45:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:45:06.559+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:2021-07-10T17:50:06.560+0000 7f9d06e5c700 0 monclient(hunting): authenticate timed out after 300 2021-07-10T17:50:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2021-07-10T17:50:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: 2021-07-10T17:50:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 2021-07-10T17:50:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2021-07-10T17:50:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2021-07-10T17:50:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cluster1 mirror images 2021-07-10T17:50:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 'cluster1 mirror images' 2021-07-10T17:50:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2021-07-10T18:40:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T18:40:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: listing images failed: (110) Connection timed out 2021-07-10T18:40:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: 2021-07-10T18:40:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cluster1 mirror mirror pool info 2021-07-10T18:40:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 2021-07-10T18:40:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 'cluster1 mirror mirror pool info' 2021-07-10T18:40:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2021-07-10T19:30:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T19:30:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 2021-07-10T19:30:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: 2021-07-10T19:30:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout:cluster1 mirror mirror pool status 2021-07-10T19:30:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 'cluster1 mirror mirror pool status' 2021-07-10T19:30:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ CEPH_ARGS= 2021-07-10T19:30:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2021-07-10T20:20:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:rbd: couldn't connect to the cluster! 2021-07-10T20:20:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stdout: 2021-07-10T20:20:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:++ echo 2021-07-10T20:20:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi105.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls 2021-07-10T21:07:01.638 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-07-10T21:07:01.643 DEBUG:teuthology.task.console_log:Killing console logger for smithi062 2021-07-10T21:07:01.644 DEBUG:teuthology.task.console_log:Killing console logger for smithi105 2021-07-10T21:07:01.644 DEBUG:teuthology.exit:Finished running handlers